~2017.4.17 UpdateThe Urbit team
Here's our April update, covering the last month or so. I've taken people's own reports and added a little background where necessary.
It might help to have a broader roadmap. We'll post that next.
I've been focusing entirely on
After documenting most parts of the existing talk implementation, I tore it apart into a "guardian" and "agent" (daemon and user-agent), or "broker" and "reader" for talk specifically.
Along the way, I took note of and corrected some architectural weirdness, with the prime goal of making the whole simpler and more clear. The split talk currently mostly works, but there's still some improvements to be made, both in existing implementation and in new features.
Background: in the cc-release world, the user-level application container %gall
is scheduled for a somewhat cosmetic internal partitioning. Right now there is
only one kind of "app," ie, dynamic core managed by the
%gall vane. In future
there are "guardians" (system daemons), "agents" (user agents), "tunnels" (HTTP
security drivers, currently in %eyre), and "gateways" (foreign API interpreters).
This is not as big as a change as it sounds -- all these partitions use the same
mechanisms. It's just about policy limitations. Eg: only a guardian can receive
inbound messages and subscriptions. In
:talk, the separation matches the split
between mailserver and mailreader in classic Unix email, news server and news
reader in NNTP, etc.
I've started a new multi-part tutorial series on building and deploying Urbit web apps. The goal of the series is to highlight Arvo's technical features and the flexibility of Urbit's API/UI model through interactive code examples and full demos that technical readers can extend.
Background: Urbit isn't yew a platform you'd want to deploy anything real on. But userspace isn't going to change that much, so we're trying to make it more accessible for early explorers.
It's true that the network has now maintained continuity for almost nine months, while ~zod has eaten 180GB of of events and seems no worse for the wear. This should not be mistaken for stability! Any data in your present urbit will need to be retrieved by hand when the universe is finally rebooted. And of course, the PKI is still non-final and invalid.
And the changes to the core
%gall programming model will be mostly cosmetic.
%gall will do more to help you handle subscriptions; pending events
will be inside your core; little stuff like that.
The main change for the programmer is just that in the present
%ames (whose total rewrite is now being tested), message transaction handling is
shitty from end to end. But you're used to that. Distributed programming with
exactly-once transactional messages is such an awesome experience that even when
it's broken, it's kind of ok.
Before 0.5, Urbit ran as a single monolithic process controlling everything from terminal I/O to computing and storing events. In 0.5, Urbit is split into a more traditional database-server architecture. Thanks to a wrapper application, casual users don't have to notice much difference if they don't want to - except that the bottleneck in the old event code is gone, so things are a lot faster. But the new architecture has a lot more flexibility.
At the top level, there is a single urbit daemon process which runs in the
background, handling I/O. The daemon can govern multiple ships, each of which has
its own worker process to compute and record events. Terminal I/O is moved to a
separate client process, which speaks a noun-based protocol to the daemon. The
sole client isn't the only thing that can speak this protocol - there's also a
command-line client that can handle system admin tasks like loading piers, and
urbit commands like the current
urb.py. You can also write your own clients if
you're adventurous, though
libnoun doesn't quite reach the level of a stable API
Background: the "bottleneck" is that old event code processes its full input queue before deigning to emit any outputs. This highlights Urbit's chelonian agility with brilliantly perverse moments of inexplicable latency. We also have no confidence whatsoever in the data integrity of the old storage layer.
The new Urbit looks much more like a regular production server. Everything outside the noun library has been replaced. We're also hoping to add FUSE access to the Urbit namespace, though the priority isn't immediate.
I am in the middle of making an architecture change to the console protocol, with the eventual goal of being able to reconnect sessions.
Background: the console protocol was in a sense the initial test case for Urbit as a true OS. If you've built a working command line by passing events between separate input, history, and application subsystems, with correction/completion that requires multiple writers to the input buffer, you've built an OS.
On the other hand, your initial test case rarely ends up as a showcase demo. For
instance, one of the things we didn't understand at first was the importance of
the CQRS pattern (command-query separation) behind Urbit's separation of
%peer. Pokes (messages) are one-way commands, not queries. Peers
(subscriptions) have no effect on the server state.
The relationship between pokes and peers is particulary delicate in a very interactive application, like a console. Proper CQRS dictates that sending input to a virtual console, and subscribing to its updates, should be totally orthogonal. But needless to say, this is not how we originally did it.
If you remember the last update, we are obviously asking our console for a lot of
UI power. We are also redesigning
:talk to use it even on the Web. But we have
to fix the protocol first.
I'm putting together new materials for the public.
I've done two main things: adding formal decorations to Hoon, and making the Arvo boot sequence coherent.
The coherent boot sequence puts a complete Urbit installation in a single sequence
of initial events. As always in the development branch, this is a true boot
sequence which proceeds directly from nock. We no longer use the
-A flag to
load the initial filesystem from an Arvo repository, nor do we boot userspace over
the network. Treating the whole install as a single noun is very helpful in
ensuring consistency for the developer. (We're aware that distributing source
files jammed into a binary pill isn't optimal from the Internet's perspective --
the only inscrutable content actually in the event stream is the self-compiling
compiler. A fancier interpreter should probably assemble its own boot sequence
from a transparent resource directory, not a binary pill.)
For formal decorations, see the accompanying forum post. Briefly, I started from
an initial draft by
~ponnys-nacwer and added a lot of elaborations. The goal is
to get Hoon to a high level of literate programming, which documents not only APIs
but also data structures. Integrating documentation into the type system is
especially important on a user-friendly system with long-lived typed data
structures. Hoon is now at 144 Kelvin.
The parser and AST changes required for decorations also helped me find a way to disambiguate direct and indirect Hoon syntax, eliminating the duplicate irregular syntax introduced in Hoon 151 -- if we want. Backward compatibility was almost perfect, requiring only a few touchups across the whole codebase.
This actually went up on
/fora before anything else. Feel free to chime in on
this thread with
comments and questions. We're here to answer them!