Urbit / Posts

~2016.8.30 Update

Here's August's monthly letter from Tlon, only slightly late. We've been doing a lot of planning and, unfortunately, a little bit of research.

We hope we're done with this research stuff. There is really nothing more unpleasant than CS research. But the protocol and crypto state (%ames and %jael) needed another round of basic architecture.

Infrastructure milestones

By “infrastructure” we roughly mean everything in /===/arvo, ie, the kernel and vanes.

Infrastructure development in the presently foreseeable future is a system of four milestones, the first of which we've hit. We have a reasonably good start on the second:

  • FI - First inception: The first milestone (August 15) was “first inception” (FI), where we claimed our true and final urbits.

  • CC - Continuity candidate: The next milestone is a “continuity candidate” (CC), a release where it's at least possible that we'll need no further continuity breaches and/or reinstalls. CC requires a nontrivial amount of further engineering work.

  • FC - Final continuity: Sometime after that comes “final continuity” (FC), where we explicitly renounce continuity breaches. CC to FC is a testing milestone and is satisfied by collective judgment.

  • SF - Security freeze: The last milestone is “security freeze” (SF), where we declare the PKI final and issue live keys.

FI: first inception

The outcome of the FI milestone was a working network, with multiple federated stars and final @p names. (Modulo some justified complaints about the lut suffix.) Federation probably means we can invite a lot more people to Urbit and it'll keep working, but let's still be a little cautious.

The FI keys are not final, but the cryptosystem is (though there may be still be bugs in our implementation).

The Urbit network works and is an adequate, if still quite buggy, social network, with two modes of social communication: chat and long-form text with comments.

People can learn to program in Hoon and even write %gall apps, but the documentation is lacking and involves wide gaps, which must be filled by reverse engineering or public assistance. Fortunately the Tlon people are reputed to be quite helpful.

At least one breach is certain. All data in your FI urbits will be destroyed! In the breach to the CC release, we will port the address allocation registry (the urbit.org womb) manually to the new network.

If you have been spawning planets from your stars, you'll need to follow our procedures, which we'll post before the next breach. Emergency breaches may also happen. We recommend against spawning at the moment, but of course your star is yours.

Data in FI urbits should also be considered completely insecure. (In fact, the urbit process probably should be run in a VM if there is any other important data on the machine, since we have not done any real security analysis and network data is used.)

CC/FC: continuity

The CC release at least attempts to maintain continuity. What is continuity?

Continuity is when we stop rebooting the universe. You can leave a continuous urbit turned off on a USB key in your attic for 10 years. When you activate it on in 2026, it will probably spend several days upgrading itself, but it will work.

CC is just a candidate. FC is when we actually believe we've achieved continuity.

Here are some features we've lumped into the CC/FC bag. This list may expand, but hopefully won't contract.

Urbit is a reliable datastore. For instance, as a developer, you should edit your source files in the Urbit filesystem %clay, then back them up with git via a one-way mirror. (Most people now edit outside Urbit and copy trees in.) Of course, %gall application state you create in a FC urbit can and should also last forever, via state adapters.

Urbit has a reliable storage system that can use a cluster log like Kafka, does not lose data even on Linux, encrypts the image as well as the log, and allows log pruning.

Urbit handles all crashes correctly on all platforms. A stack overflow or interrupt always produces a usable trace.

Urbit boots with the proper, beautiful, functional, boot sequence defined in the whitepaper. It's single-homed -- designed to operate one ship per kernel -- and uses the event security model from the whitepaper.

Urbit has a working end-to-end “permanent networking” message system with in-order, exactly-once delivery and end-to-end acks, and perfect causal consistency over the wire with no head-of-line blocking between different ducts.

Urbit no longer mixes cryptosystem code with networking code. Secret storage moves into a separate %jael vane. %jael will hold public, private and symmetric Urbit keys, foreign API keys, tokens, passwords, etc; and other capabilities, like Urbit invitations.

Urbit has a communication recovery procedure so that a brain-damaged ship can resume communication after it loses information, even if all it knows is its own identity and key. Only-once delivery is violated, but the onus for recovery is placed on the damaged ship rather than its healthy neighbor.

If a ship has a static IP, Urbit can serve a secure website without proxying, either at its own urbit.org subdomain or at a private domain.

Urbit has have a FUSE-oriented filesystem interface to the local OS, rather than using libuv filesystem monitoring.

Urbit has a mark system that supports mark containers, and handles mark evolution properly in both the network and the filesystem.

Urbit handles library versioning properly in %ford, never using => to load libraries, using aliases correctly to allow multiple conflicting versions in a dependency graph.

Urbit has a stable, extensible but backward-compatible I/O protocol between Arvo and vere (the Unix process). Any future Arvo will work on an old vere.

Urbit uses Murmur32 rather than FNV as a short hash (mug).

Urbit is still not a cryptographically secure system, and doesn't allow true cryptographic ownership transfers. The cryptographic transfer features exist, but using them is a bad idea until the keys are final.

(For right now, everyone who issues urbits to others needs to at least track ownership centrally, ideally with a mirror database outside Urbit. We could not stress this more stressfully!)

Urbit has received a thorough performance and scaling audit, and there are no trivial efficiency problems to solve.

The Urbit console can output colored text and control a 2D ASCII text array (curses style). The web console (not web dojo) works with some keylag.

All Urbit vanes are thoroughly documented/specified with an overview document and inline comments.

SF: security freeze

The SF release will render Urbit a “live” cryptographic system. It is unnecessary to point out that all subsystems in the security chain must be audited, and operational procedures for stars and galaxies need to be standardized.

(Yes, the SF release needs to invalidate the current cryptographic structure, even if it hasn't been compromised. Since it's not secure, we don't know it hasn't been compromised.)

However, the SF release keys will be instantiated via network update, which is essentially “god mode.” New keys will be distributed to old key holders, and any disparities between cryptographic and centralized ownership records will be resolved in favor of the latter. Before SF, no one can gain anything by stealing a key or transferring it without authorization.

A software update can of course do anything to the PKI. After SF, at least, global Arvo updates (which ideally can become rarer and rarer) will need to be approved through a constitutional, republican governance process.

After the SF release, we have no control over cryptographic transfers, but the Arvo code itself may penalize long wills to minimize velocity and keep Urbit from acting as a currency. (This is essentially a Tobin tax.)

The SF release needs to run the actual Arvo interpreter, with jets, in a completely sandboxed subprocess with no I/O privilege. Sandboxing jets, or even a safe jet language, should also be investigated. Network jet update must also be possible.

An SF urbit can issue and manage children without any problem. It can be hosted easily in a commercial data centre. The interpreter can migrate images securely, automatically and efficiently to a different hosting service.

An SF urbit will have a framework for DoS defence, blocking and banning, and other nuisances of the modern world. There will be at least the beginning of a global reputation system.

CC: work overview

The one subsystem that needs a complete rewrite is %ames (the network). The %jael vane is completely new. Fortunately, the new %ames vane compiles, and the %jael interface is defined (and not that %jael can be complicated).

The mug replacement work is also done, but needs to be debugged (which is no easy task).

Applying the canonical boot event system sounds like a giant change, but it's actually pretty straightforward -- same for single-homing. Arvo proper is only a few hundred lines of Hoon.

The event system in vere needs to be replaced, but we have a module built by a contractor with both the local FS and Kafka.

All the vanes besides %ames need documentation and a general tuneup, but this should not rise to the level of a rewrite for any other vane.

Community involvement

Urbit's most serious problem right now is that many more people are interested in it than are actually working on it. This reflects its heritage as a basement project.

Our most important work item remains opening up the codebase so that community development can happen. We'll start turning the CC tasks into community operations as fast as possible.

But this is a separate topic. It fits under a discussion of Tlon's corporate plans and operations, which we'll post later.

Contributors

The contributors have been amazing. We need to make more use of their talents, which are growing daily. And we need to do a better job of chronicling their deeds! But for now, github remembers.

Keep in touch / office hours

:talk is always a great place to come and ask questions. Going forward someone from Tlon will be available in :talk from 12PM - 3PM PDT each weekday.

Post to our new forum! Especially this thread to discuss this very post. Don't be afraid, the water's fine! Actually there are a few glitches, but it's hosted on Urbit (and also backed up). For the less daring, r/urbit is a reliable old standby.

For anything else you can always get in touch with us via [email protected].