~2016.10.06

~2016.10.6 Update

September's monthly update slipped into October! But don't worry, it's a good one.

"Hard tech"

In this amusing talk, Chad Rigetti defines "hard tech" as "solving problems you don't even know are solvable." Rigetti is building a quantum computer, which is definitely "hard tech." More power to him!

But software can be "hard," too. Is Urbit "hard tech"?

The correct answer at this point is: no, but it used to be hard tech. Now it's actually real, if not exactly easy. The Urbit network hasn't crashed once this summer, though we did one intentional breach (right before delivering the stars). I'm still kind of amazed by this.

But we still need to get to the point where (a) the whole network can't crash ever, (b) you can put your data in an urbit and be pretty sure it'll live forever, (c) Urbit works well as a social network, and therefore (d) we can distribute planets to ordinary human beings and not have them be like, "gah, what is this?"

If we had the resources of Microsoft or Google, or even of Mozilla, we could solve these problems with a brute-force human-wave attack. We could just debug the system into existence. It must be nice.

But not. In some ways it's essential to Urbit's mission that there isn't a giant army behind it. It means that the only way Urbit can succeed, or even survive, is to be excellent. That's the right attitude for a stack designed for the 2020s. Not that we can wait until the 2020s to get (a) through (d). Just that we don't want our children and grandchildren to curse the day this stack was born. Unix, the Internet and the Web gained the world and lost their souls -- we don't have that option.

Given this attitude, the thing that terrified me most of all about this summer's crowdsale is that the most important parts of Arvo -- the network and the PKI -- simply didn't exist in any viable sense. Yes, the present %ames network does net work. There is even in theory a sort of PKI, although no one has ever even tried to revoke a key. It might be secure -- who knows?

More concretely, the existing %ames vane (kernel module) contains both the protocol and the PKI, mixed together in a single giant state machine of the sort I've learned never to build again. There is no perfect code anywhere in the present production Arvo, but the only body of dead-end code -- which needs replacement, not documentation and maintenance -- is %ames.

Unfortunately, networking and key management is the most important part of Urbit. In a sense it is Urbit. So having it not exist, except as a design in our heads, was not a good thing.

While documenting, evangelizing and in general growing Urbit is vastly important, what's even more important is finishing it -- without compromising its elegance. Elegance is the only thing Urbit has. It's not like there aren't other operating systems.

So the first thing I did after the crowdsale was to disappear into a hole and take care of this technical debt. This was not "hard tech" in the sense that I knew it was solvable. I didn't know it was solvable well, though. And it has to be solved well, because otherwise who cares.

Check out our new, improved cc-release

I'm delighted to report that the new, rewritten %ames network, and the all-new %jael PKI, are now at code complete. (Code complete means "compiling but not tested," which is actually saying a lot in a typed functional language.) They've been checked in to a new cc-release branch on github.com/urbit/arvo.

I think these two vanes are definitely the best code in Urbit. There's still a bit of fundamental work in Arvo (implementing the boot sequence and event security mechanisms described in the whitepaper), but this is not "hard tech" -- the designs are clear and surprises seem unlikely.

%xmas: the new network

New %ames (currently arvo/xmas.hoon, since old %ames is still in the build) has had all the key management stripped out; it's just a network vane. It's also no longer a single giant state machine, but a system of five simple 200-line state machines.

It was wonderful to discover that Hoon lets you write purely functional programs with what are essentially global variables. Specifically, nested cores generalize across global variables, local variables, and objects, in a way I find technically quite lovely if I do say so myself. On the other hand, Hoon ain't magic, and it's not like we've solved any of the software-engineering problems inherent in a 2000-line state machine based on global variables. So, nice to kill that.

Don't look at new %ames just yet. It was done in July and August. It needs a few touchups to fit the %jael interface. And it needs to be brought up to %jael's quality standards.

%jael: reference code, PKI, and personal ledger

%jael (/arvo/jael.hoon), the PKI, is what I'm really excited about. It's what you should take a look at if you know any Hoon or even if you don't. There's several really important things about %jael.

The first is that it's actually reference-quality code. Or so I believe. It's literally intended to serve as a reference style for what all production-quality Hoon should look like. If you're reading the code and you see anything that isn't absolutely perfect, whether it's cosmetic or structural or anything in between, file a PR. %jael is fully commented -- if you think it needs more comments, or fewer, file a PR...

Needless to say, a reference style will be an enormous help in managing community contributions. It's not that, for now at least, we won't accept patches not written in line with the reference %jael style. But we'll always accept patches to bring 'those patches' up to reference. For the moment, the code will just define the style by analogy, but it'll eventually spawn an actual document.

If you're feeling ambitious and want to change the reference style itself, your change is best submitted as a patch to %jael itself. And really ambitious people can create their own alternative Hoon styles -- hopefully defined at the same same level of precision.

Second, %jael actually implements the Urbit PKI as designed, right down to unusual but politically critical features like escaping a bad parent. Since the code is readable and heavily commented, it can and will serve as the best possible reference for the PKI's semantics.

Both %ames and %jael are designed for the new and correct single-homed urbit model, where one pier is one ship. In the new world, to boot your planet, you use your ticket to generate a symmetric key that authenticates you to your parent. When you share your public key with your parent, over that initial secure channel, it notices that it hasn't signed it yet and shares its signature back to you.

Third, %jael isn't just a PKI -- it's a generalized system for formal promises. The whole point of a PKI is that entities can make secure statements to each other. Of course, every :talk message is a signed statement, but the really important messages are promises -- such as, for instance, "Tlon owes you 3 stars."

(Right now, the stars we sold in the crowdsale are recorded in the state of a higher-level application (:womb). (We have an encrypted offline backup, of course.) This is fine at a certain level, but it's a bit rickety and not really up to Urbit's standards of architectural elegance. Promises and secrets are fundamental state and need to be handled at a fundamental level.)

In a secure distributed social network, people need to be able to make formal promises to each other. Essentially, every ship is an accounting entity. It records its liabilities, which are the promises it's made to others, and its assets, which are the promises others have made to it. When I update the promises I've made to you, I send you an updated statement.

All kinds of promises fit this mold. For instance, while you don't usually think of a symmetric key as a promise from one entity to another, it is. It's a promise from ship A to ship B, that B can identify itself to A by this secret.

So %jael will, among other glorious things, essentially allow every galaxy to serve as an exchange for star invitations, and any star to serve as an exchange for planet invitations. Mere payments are just the start of what you can do with this. And of course, one of our big near-term goals is to make first Urbit invitations, then (once the crypto is truly final) actual ships, tradeable on "regular" cryptocurrency exchanges.

One of my views about Bitcoin is that its decentralized, trust-free ledger isn't really the most important thing about it. Humans build trust quite easily. For instance, Bitcoin doesn't even enable trust-free payments -- a payment is always an exchange for other good or service. Do I provide the good or service before, or after, the Bitcoin payment? Either way, trust is unavoidable.

The important thing about Bitcoin is just that it enables users to manage their obligations to each other digitally, without interacting with a 20th-century asset management system made of paper and magnetic tape. Giving every user their own asset management ledger is another small revolution in that direction. And if I'm defining and issuing my own digital assets, why do I need to put them on a global blockchain? You're still trusting me to actually live up to my promises.

Last but not least, %jael solves one of Urbit's most important and hardest problems: how do I back up my urbit? I know I said this was impossible, but I lied.

Urbit uses persistent sessions, which means a pair of ships uses the same session sequence numbers for life. Since every packet is a transaction, when you send an acknowledgment, you mean it. The upside is that we get exactly-once messaging, which if you've ever done distributed programming is a very, very good thing. The downside is that if you lose your state, there's no way to recover it, since you don't know what sequence number to send next. (See under: Two Generals' Problem).

In the new world, the Unix layer will subscribe to a stream of %jael changes which contain all certificate and rights actions. We'll save this separately from the rest of the pier state, and it's small and trivial to back up.

When restoring, if you have all your private keys but have lost all other state (obviously if you don't have the latest private key, you're hosed), you sign a new version of your will which contains a "brain damage" bit. When this will is propagated to your neighbors, they learn that you're brain-damaged and reset their sequence numbers -- and forget you know anything they told you, treating you as a new stranger.

At the application level, all your subscriptions get closed. Apps already have to deal with this. Forward messages shouldn't involve any state coupling; that's what subscriptions are for. In case there is some coupling anyway, you'll get an event that "ship X lost his mind, forget anything you know about him."

Essentially, we're cheating the fact that you can't have transient sessions and exactly-once delivery by letting the abstraction leak in this special case, and putting the responsibility for fixing it on the brain-damaged ship. It may even leak up to the user. But dude, you're the one that let your brain fall out of your skull.

Another thing I like about this solution (and about %jael in general) is that it solves the problem of how you use someone else's secrets, if you manage to steal them. There's no point in security through technical difficulty -- if you have the latest private key for a ship, you're the captain now. You shouldn't have to hack something up to make this work.

Keep in touch / office hours

:talk is always a great place to come and ask questions. Someone from Tlon will be available in :talk from 12PM - 3PM PDT each weekday. And our forum is a great place to post questions. We've created this thread to discuss this post.

For anything else you can always get in touch with us via urbit@urbit.org.