~2017.4.17 Update

Here's our April update, covering the last month or so. I've taken people's own reports and added a little background where necessary.

It might help to have a broader roadmap. We'll post that next.

~palfun-foslup (Fang)

I've been focusing entirely on :talk work.

After documenting most parts of the existing talk implementation, I tore it apart into a "guardian" and "agent" (daemon and user-agent), or "broker" and "reader" for talk specifically.

Along the way, I took note of and corrected some architectural weirdness, with the prime goal of making the whole simpler and more clear. The split talk currently mostly works, but there's still some improvements to be made, both in existing implementation and in new features.

Background: in the cc-release world, the user-level application container %gall is scheduled for a somewhat cosmetic internal partitioning. Right now there is only one kind of "app," ie, dynamic core managed by the %gall vane. In future there are "guardians" (system daemons), "agents" (user agents), "tunnels" (HTTP security drivers, currently in %eyre), and "gateways" (foreign API interpreters).

This is not as big as a change as it sounds -- all these partitions use the same mechanisms. It's just about policy limitations. Eg: only a guardian can receive inbound messages and subscriptions. In :talk, the separation matches the split between mailserver and mailreader in classic Unix email, news server and news reader in NNTP, etc.

~tonlur-sarret (Keaton)

I've started a new multi-part tutorial series on building and deploying Urbit web apps. The goal of the series is to highlight Arvo's technical features and the flexibility of Urbit's API/UI model through interactive code examples and full demos that technical readers can extend.

Background: Urbit isn't yew a platform you'd want to deploy anything real on. But userspace isn't going to change that much, so we're trying to make it more accessible for early explorers.

It's true that the network has now maintained continuity for almost nine months, while ~zod has eaten 180GB of of events and seems no worse for the wear. This should not be mistaken for stability! Any data in your present urbit will need to be retrieved by hand when the universe is finally rebooted. And of course, the PKI is still non-final and invalid.

And the changes to the core %gall programming model will be mostly cosmetic. For example, %gall will do more to help you handle subscriptions; pending events will be inside your core; little stuff like that.

The main change for the programmer is just that in the present %arvo and %ames (whose total rewrite is now being tested), message transaction handling is shitty from end to end. But you're used to that. Distributed programming with exactly-once transactional messages is such an awesome experience that even when it's broken, it's kind of ok.

~ramtev-wisbyt (Raymond)

Before 0.5, Urbit ran as a single monolithic process controlling everything from terminal I/O to computing and storing events. In 0.5, Urbit is split into a more traditional database-server architecture. Thanks to a wrapper application, casual users don't have to notice much difference if they don't want to - except that the bottleneck in the old event code is gone, so things are a lot faster. But the new architecture has a lot more flexibility.

At the top level, there is a single urbit daemon process which runs in the background, handling I/O. The daemon can govern multiple ships, each of which has its own worker process to compute and record events. Terminal I/O is moved to a separate client process, which speaks a noun-based protocol to the daemon. The sole client isn't the only thing that can speak this protocol - there's also a command-line client that can handle system admin tasks like loading piers, and urbit commands like the current urb.py. You can also write your own clients if you're adventurous, though libnoun doesn't quite reach the level of a stable API yet.

Background: the "bottleneck" is that old event code processes its full input queue before deigning to emit any outputs. This highlights Urbit's chelonian agility with brilliantly perverse moments of inexplicable latency. We also have no confidence whatsoever in the data integrity of the old storage layer.

The new Urbit looks much more like a regular production server. Everything outside the noun library has been replaced. We're also hoping to add FUSE access to the Urbit namespace, though the priority isn't immediate.

~pittyp-pittyp (Anton)

I am in the middle of making an architecture change to the console protocol, with the eventual goal of being able to reconnect sessions.

Background: the console protocol was in a sense the initial test case for Urbit as a true OS. If you've built a working command line by passing events between separate input, history, and application subsystems, with correction/completion that requires multiple writers to the input buffer, you've built an OS.

On the other hand, your initial test case rarely ends up as a showcase demo. For instance, one of the things we didn't understand at first was the importance of the CQRS pattern (command-query separation) behind Urbit's separation of %poke and %peer. Pokes (messages) are one-way commands, not queries. Peers (subscriptions) have no effect on the server state.

The relationship between pokes and peers is particulary delicate in a very interactive application, like a console. Proper CQRS dictates that sending input to a virtual console, and subscribing to its updates, should be totally orthogonal. But needless to say, this is not how we originally did it.

If you remember the last update, we are obviously asking our console for a lot of UI power. We are also redesigning :talk to use it even on the Web. But we have to fix the protocol first.

~ravmel-ropdyl (Galen)

I'm putting together new materials for the public.

~sorreg-namtyv (Curtis)

I've done two main things: adding formal decorations to Hoon, and making the Arvo boot sequence coherent.

The coherent boot sequence puts a complete Urbit installation in a single sequence of initial events. As always in the development branch, this is a true boot sequence which proceeds directly from nock. We no longer use the -A flag to load the initial filesystem from an Arvo repository, nor do we boot userspace over the network. Treating the whole install as a single noun is very helpful in ensuring consistency for the developer. (We're aware that distributing source files jammed into a binary pill isn't optimal from the Internet's perspective -- the only inscrutable content actually in the event stream is the self-compiling compiler. A fancier interpreter should probably assemble its own boot sequence from a transparent resource directory, not a binary pill.)

For formal decorations, see the accompanying forum post. Briefly, I started from an initial draft by ~ponnys-nacwer and added a lot of elaborations. The goal is to get Hoon to a high level of literate programming, which documents not only APIs but also data structures. Integrating documentation into the type system is especially important on a user-friendly system with long-lived typed data structures. Hoon is now at 144 Kelvin.

The parser and AST changes required for decorations also helped me find a way to disambiguate direct and indirect Hoon syntax, eliminating the duplicate irregular syntax introduced in Hoon 151 -- if we want. Backward compatibility was almost perfect, requiring only a few touchups across the whole codebase.

Oh, and

This actually went up on /fora before anything else. Feel free to chime in on this thread with comments and questions. We're here to answer them!