Urbit / Posts

~2017.5.11 Roadmap

Urbit today (May 2017) is a working proof-of-concept. The Urbit test network has been up continuously since last August, hosting its own distributed chat, distributing its own updates, serving its own website and forum, etc, etc.

Everyone has heard of the minimum viable product (MVP). Urbit in summer 2016 (the current test network) is the MVP's first cousin: the AVP (almost-viable product).

From AVP to MVP

Urbit is an open-source project. Its sponsor, Tlon, is a company. So we can't exactly tell the public to go away. But there is nothing much the public can use Urbit for, besides hanging around and talking about Urbit.

Going from AVP to MVP means changing that. And of course, it means issuing real Urbit keys. And promoting Urbit to users. And preaching it at developers. And all that normal stuff. But not until we're ready, because we ship only the most beautiful bits.

Sometimes we forget to remind people that no one has ever built anything even remotely like Urbit. It's not like, people had cars, so we built a better car. It's like, people had cars, so we built a rocket unicycle.

Our rocket unicycle is a lot more awesome than a Chevy. But it demands more custom engineering. We have to invent every little widget as we need it. Usually that involves rebuilding it two or three times (see the (see the long-term strategy). This process converges from the bottom of the stack up, and accelerates as our progress advances.

Nock, which is now frozen, and which is a page of pseudocode, took from 2002 to 2008 to invent. Hoon took from 2008 to 2012, and is stable but not frozen. All the real technical problems in 2016 Urbit are either in the top layers (Arvo, its vanes and apps), or outside Urbit proper (Vere, the Unix event processor).

Except for one major component (the security vault %jael), there isn't any single big thing wrong with 2016 Arvo or Vere. In retrospect, they can even be seen as inspired stabs in the right direction. But hammering square pegs into round holes isn't the Urbit way.

If we made our AVP into an MVP by straight-out debugging, we'd be limping across the finish line. An MVP must be minimal in affordances. In its aesthetics, it must be sublime. It has the rest of its life to grow features, but beauty only fades.

A parallel process

Architecture is entropy in reverse. Urbit has to become more elegant, as it becomes more functional. This delicate process must be carefully controlled.

To keep Urbit on the path to absolute zero, we have to work in three directions at the same time:

  • Stabilizing the Urbit infrastructure

  • Developing the Urbit user experience

  • Documenting the Urbit system

Pushing in all three directions at once keeps the whole project in balance. We could drop any two of these goals and head at top speed for the other. This would produce a nice simulation of doneness in a single dimension.

But if we just hardened our infrastructure from the bottom up, we wouldn't be responding to the needs of the user experience. If we just built a user experience on top of our prototype infrastructure, it wouldn't be stable enough to be usable. If we documented the prototype, it would become much harder to revise. And so on.

And the balance between these directions shifts with the seasons. After spending the fall and winter writing infrastructure, and the spring testing it, it'll be time to spend summer and fall working toward the user and the developer.

Since the last crowdsale

Maybe this is too vague. What we actually did since the last crowdsale:

We replaced about half the Unix interpreter (the whole vere directory, event handing and storage), and about a third of Arvo: the lifecycle function, boot sequence and kernel %arvo, the network %ames, the PKI (adding a new security module %jael), the console :drum, and the :talk app. If Arvo was a car, we replaced the powertrain.

At the Hoon level, we added formal docstrings and reorganized (or rather, organized) the kernel and standard library. At the Nock level, we of course did nothing. And on the client side, we redesigned the Web API, and prototyped a completely new Urbit user experience. And all the new code is high-quality, fully documented, and generally ready for anyone to copy and/or abuse.

This cc-release (continuity candidate) isn't quite done — the Arvo code is written but not fully tested, and one module (the application sandbox %gall) is still waiting for its rewrite. But the new Arvo has that “not quite perfect, but basically right” feeling of architectural doneness which Hoon has had for a while.

Note that we remain a super-tiny team. We've had some serious outside contributions, but clean loose ends that outsiders can pull on remain frustratingly scarce. Hoon is too stable and Arvo is too unstable. Architectural work scales poorly in personnel, and you can't really renovate a house while people are trying to live in it.

Moving forward

After cc-release this summer, we'll move from bottom-up, infrastructure-driven engineering, to top-down, user-driven engineering. Naturally this is dd-release (deck-directed).

Our planning target is a stable, useful Urbit we can ship in 2017. Since we're working top-down, we can abandon the deep architectural essays and work, like normal human beings, from an illegible list of features scribbled on a poorly-erased whiteboard.

A whiteboard list

Our feature list for dd-release, in no particular order:

  • The top-level :deck UI, an interaction multiplexer like GNU screen, but for terminals or web consoles. (The front end of :deck is a working prototype now.)

  • The “helpful console,” which offers contextual help and lets you construct any dojo command with a wizard-style UI. The three-way combination of a command parser, contextual completion and help, and webform input, may be our most interesting little challenge. We think we know to do it.

  • Useful, well-designed web API integrations which contributors can easily clone to extend Urbit's reach. Stars can proxy APIs for their planets, metering free service.

  • Integrations for email and SMS are live, so that any ship can at least message its owner. A storage integration to S3, etc, lets us save external blobs. A DNS integration lets us forward ship.urbit.org to ships with public IPs.

  • Real keys are ready to be born. An instantiation protocol for the real Urbit keys exists and has received some public review. The current options: physical delivery of tamper-resistant artifacts, or registering ownership on Ethereum or another blockchain. (We'll release a separate essay on this question soon.)

  • All key updates at the planet level or above are propagated hierarchically across the network. An offline key can block all transfers. Planet owners can invite their friends to a self-hosted conversation with webmoons (guest accounts).

  • The %eyre HTTP server is cleaned up and its API simplified. Web programmers see a clear, CQRS-based query and subscription interface in JSON. A cookie makes your browser into a virtual moon, with a unique causal identity for each page instance.

  • No user operation can damage the ship, which can be reset to factory condition with just the private key.

  • Personal urbits on a home computer don't lose data, except in case of system/OS failure. Mutual exclusion on interrupt handlers is perfect. Timeouts, crashes, and user interrupts are handled perfectly. Snapshot handling is asynchronous and uses two A/B images.

  • Urbits are configured to rotate logs by default, so they don't grow infinitely. All caches have continuous expiration, and out-of-memory can't sink a ship. The Web server is secure and has adequate performance without an external cache or any auxiliary servers.

  • Urbit can connect its event loop to Kafka or a similar event replication service. Replicated urbits on a hosted service don't lose data, except through acts of God or government.

  • One star on a normal virtual host can comfortably manage at least 256 active planets. Dormant planets consume minimal local or remote resources.

  • The whole Urbit stack is managed with telescoping Kelvin numbers. Upgrades succeed perfectly or crash the transaction. The stack is permanently upgradeable, even for backward-incompatible changes in Hoon.

  • Hand maintenance is only needed to upgrade the Unix interpreter, vere. The owner gets a notification when an obsolete vere blocks an Urbit update.

  • The new security masks are respected across all vanes. The revision-control system %clay has permission masks. Untrusted code can be installed as a gateway to a foreign service with same-origin network permissions.

  • Backward-incompatible changes in the Hoon language or Hoon libraries are unlikely. But the Arvo upgrade mechanism can make arbitrary Hoon changes, compatible or not.

  • All major Arvo subsystems are architecturally stable. Namespaces, cards, URL routes, the boot sequence, and the upgrade model, are well-specified. Further change in these specs will involve a slow-track community process. But the definitions are stable enough to be taught.

  • Architectural overviews of all major Arvo components are attached to the source code. Tutorial documentation for both Hoon and Arvo is good enough to lead a naive programmer up through building a web API gateway (server side Hoon) and a web agent (client side JS).

  • Urbit stores and manages all Urbit-related development metadata: todos, bugs, releases, PRs, discussion, etc. It routes this metadata through :talk and logs it to %clay.

  • A precise system coding convention is in place and enforced, if only by measurement. All code in Urbit is graded A, B, or C. Grade A Hoon (“Prime”) is well-structured and well-explained. Grade B (“Select”) is well-structured, but not well-explained. Grade C (“Choice”) is still great code, but its style or technique may not meet our latest standards.

  • All code in Urbit except ++ut (the compiler itself) is grade B. (We can relax standards on the compiler because it tests itself so thoroughly.) All code on the Arvo powertrain (%arvo, %ames, %jael, %gall, :dojo, :helm) is grade A.

If you know the current maintenance Urbit, you know that most of these features are mostly in sight. None of them is exactly quite complete. None of them is anywhere near a stretch. We have significant parts of all of them.

Also, since this is the release after the release we're doing now, we're probably missing as many checkboxes as we've caught. And triage is a thing that happens. It's really better understood as the kind of work we're going to do.

We'd really like to get all these things done in 2017. But a lot depends on what kinds of resources we have access to. And of course, as we document the system, it gets easier for all the people who believe in Urbit to help us build it.

As usual, comments are on fora. Check out this thread.