~2017.1.30 Update

TLDR: a real Urbit is an Urbit that does real work. This means nontrivial progress in two areas: systems maturity and user experience. In 2H 2016 we cleaned out most of the worst holes in Arvo and vere, and totally reconceived the user experience. The new Urbit feels like an MVP and we can't wait to ship it.

In 2013 Urbit was intriguing. By 2015 it had become interesting. In 2016 it was actually exciting (if not always in a good way!). In 2017, we're actually planning to make the darn thing useful. (For certain values of the word "useful.")

Our cc-release stability upgrade (mainly affecting the OS layer arvo and the interpreter vere) is roughly code-complete and being tested into existence. Once it exists and is stable, it can actually be documented -- often, for the first time.

And cc-release is our first release really designed so that neither your ship, nor the whole network, can actually sink. The Urbit network had zero unplanned, and one planned, breach (network flag day) in 2H 2016. The last planned breach, the cc-release update, is planned for 1H 2017.

With the infrastructure converging, we can finally think seriously about the user experience of Urbit. What we're imagining is a small-scale social network and command-line API aggregator. It's just screenshots right now, but we're confident that we can build it and make it work.

System software is boring (at least when done right). So let's start with the user experience.

Design

Galen Wolfe-Pauly is Tlon's CEO and Urbit's interaction designer. Galen went to architecture school. He talks like it. We recorded him secretly and came away with these notes:

  • UX? As a kid it was just design or maybe web design. At some point I remember people talking about user interface or UI. But I truly don't understand where the X came from. It's like Monster Energy got involved in the branding.

  • An experience sounds like the kind of thing that's going to be littered with ads, or is going to manipulate me in some way. Something I passively consume. Not something I actively control. So I prefer interface to experience, and I think you should too.

  • Urbit has to be just as usable as any widely used app in 2017. For a personal server, the usability bar can't be Unix. A Unix server is an industrial tool. An industrial tool can't be a consumer product. You can attach a blade to the end of a drill and use it as a blender, but it's not a Vitamix.

  • What is an app, anyway? It's shared computing. Everyone's data is one data structure, in one program, on one server, owned by one corporation. With one UI for everyone to do one thing.

  • But there isn't one big corporation that writes all the code. Everyone's lives are spread across a bunch of different apps. Every app solves the ownership problem separately.

  • So the whole user experience of using Web apps involves tracking all these silly little accounts, learning each and every user experience, and glaring helplessly at each little piece of data trapped in a different silo.

  • Even your Google account doesn't feel like one thing. Whatever your Google Docs are, it's not a bunch of document files on your Google Drive. All the resources of Google can't make their Borg backend feel like one computer which belongs to you.

  • The phrase 'general-purpose computing' sounds scary, but it shouldn't be. 'Computing' doesn't mean writing code. We're computing when we browse Facebook or talk to Alexa.

  • Your personal server will be solving the same problems today's whole ecosystem of network apps solves, just all in one place that belongs to you.

  • The closest thing to a general-purpose personal server today is probably the Chinese service WeChat. If you don't know much about WeChat, you should really watch this NYT video.

  • From a distance WeChat is great. Everyone in China loves the one-app, one-identity, all-my-own-stuff user experience. It's clearly better than being forced to switch apps constantly.

  • But WeChat is just another giant shared-computing service. It's made out of the usual 2010s server stack mess. The visual and interaction design looks pretty dated. WeChat tells us that users want one experience, but not much more than that.

  • To paraphrase Walter Sobchak: say what you want about the tenets of shared computing, but at least it's an ethos. We know how to build very good special-purpose user experiences. What's the conventional UI for a general-purpose personal server? It doesn't exist.

  • For most users, your urbit runs in a data center (like a web app). You log into it with your browser (like a web app). Other than that, where do you even start? A bash prompt isn't what we're looking for.

  • But wait, is the idea of a command line bad UI? Or is the Unix command line just bad UI? We know ordinary people can use command lines. They use chatbots. Long ago, they used MS-DOS.

  • Today's computing user experiences, the PC and its mobile cousin, rejected the command line. This made computing easy enough, at the cost of hiding much of its creative power.

  • The power of a general-purpose server which can command not only local applications, but also network services, is at a whole new level. We don't think any visual metaphor or 2D control panel can do it justice.

  • We need to bring the full power of the machine to the user, without breaking their trust that the machine is on their side. We can't let the user get lost or break the system. It has to feel safe and simple to explore its furthest reaches.

  • As Neal Stephenson tells us, the command line is the natural UI for any computer. It's especially the natural UI for a network computer, like a personal server. A command line scales way better than a GUI to a large selection of actions. And a network computer can do almost infinitely many things.

  • When we've tried to put a simplified "for dummies" GUI on top of Urbit, it's just awful. It's like a bad Linux desktop GUI. You feel instantly that you've got a choice between something limited and lame, and something mysterious and scary. So we threw those designs out, in search of something better.

  • The true command line is generally thought of as something mysterious and scary. Somewhere that you can do things to irreparably damage your computer. It's powerful, but also dangerous.

  • This tradeoff — between control and safety — is false. The unfriendly command line is actually just another artifact of 70s computing. The UI of the command line is designed for a programmer, not an ordinary person.

  • bash is a bad UI because there's no easy way to control its power, or learn all the things it can do, or separate its power to do dangerous things from its power to do safe ones.

  • At the other end of the spectrum is the chatbot, which is a bad UI because it's hard to make it much more than a toy. Like so many innovations, chatbots happened by accident. It's just a way to connect an input string to network services. But the natural-language metaphor of "chat" is confining. It doesn't scale well to any problem that isn't trivial.

  • What does a great command line look like? It should have a strict syntax, not try to guess what you mean. But it can't be a constant test of your ability to memorize the phonebook.

  • Urbit already has the beginnings of a great command line. We parse your input with an actual grammar and catch errors as you type. You can't even press return on an invalid command.

  • What's missing is help. A great command line needs to reach out and hug you in every way it can. It needs to assume you don't know anything at all about the system. It's there for you to solve your problems, but also to help you explore.

  • There's actually one exotic Unix shell that does it right: fish. fish jumps through some pretty amazing hoops, like parsing manpages, to give you context and completion.

  • It's a lot easier for Urbit because we're designing the system from scratch. From a technical standpoint, all we have to do is add contextual help to our existing parser framework. An extended rule can parse a string, or tell the user what it wants to parse. In simple terms: we have to make the :dojo (our shell) much friendlier.

  • The :dojo shell is built around the concept of a generator, a program that builds a noun. The idea is that how you build the noun is orthogonal to what you do with it. You could print it, send it as a command, etc. There are several kinds of generators -- simple scripts, prompting dialogs, HTTP scrapers, etc.

  • A generator is initialized with command-line arguments and options, in the classic Unix style (except typed). In the classic Unix style, Urbit at present provides no help at all in constructing a command line. You just have to know.

  • How should it actually work? Let's say you're trying to put Urbit in control of your Twitter feed. That means you're creating a Twitter gateway on Urbit, which you can control with a |twitter command.

  • Before you type a character, Urbit offers you | (any command with side effects). (You could also add a ! prefix, which is like sudo -- it lets you do something dangerous.) Once you type |, you can page through a list of commands. Once you type |t, that list is much shorter. Once it can tab complete to |twitter, it prompts you with that.

  • And once you get to |twitter, you're in a "meta-generator" that's helping you build the correct command line. At least in a browser, you'll be stepping through a form with rich entry. But the command line will show you the text of the command you're building, so next time you can just type it.

  • Building a command is just a case of data entry. In a command-line world, data entry is always serial. You answer questions serially, one at a time. The only navigation is forward and back.

  • A great Urbit console also has to be accessible both from a browser and a Unix terminal. That means a prompt needs to tell the console if it could be a radio button, even though a terminal can't have a radio button. We'll improve the interactivity of our terminal a little, but we're not rewriting Lynx.

  • So, what do you do with this fancy command line? Command everything -- from your urbit to the world. Installing an app is a command. Publishing a tweet is a command. Paying someone in bitcoin is a command.

  • Your urbit can show you all the things you can do, and help you do them. If there's something you'd like it to do, build it!

  • A great command line alone is not all that Urbit needs. But it feels like the right central paradigm for interacting with your Urbit. We'll get into the specifics in a future post.

Infrastructure: stability, maturity, performance, documentation:

We dragged Curtis, whose hide has turned completely white like a cave fish, from his coding hole to stammer out this update:

We're actually doing pretty well on the infar. Despite there being all kinds of unacceptably broken things about Urbit 2016, the network has actually stayed up since August, and we haven't had an unintentional continuity breach since June.

Of course, it helps that we don't ask anyone to use Urbit in earnest. And ships still do sink. Since an urbit is an identity, sinking your planet is a horrendous experience. You'll never bond with your second planet the way you did with your first -- never mind any data you may have lost.

Our upcoming cc-release is the last planned breaching release. With certain exceptions discussed below, all major parts are now (January 2017) code complete or better.

This includes the rewritten network stack (%ames), code complete but untested; the new secret vault and promise tracker (%jael), mostly tested; and the new u3_pier event execution framework, with worker processes and real two-phase commit, smoke-tested. We've also completely restructured hoon.hoon and zuse.hoon, which now are at least cosmetically acceptable.

This is not to say that the cc-release Urbit won't in many ways embarrass us as engineers. It will. But your ships won't randomly sink, your secrets won't randomly leak, and your datas won't randomly rot. Or if they do, it's an implementation bug, not a design flaw.

There are still a few cracks we need to fill in. The main one: some top-level adjustments to the Arvo event system. The boot sequence now executes correctly, but the actual boot events need to be totally refactored, as does the rather critical Unix-Arvo interface. The long-promised security mask and voltage flags need to be added, and vanes adjusted to use them. Causal tracking of execution crashes needs to actually work. At the user level, :talk needs to be split into a user agent and a daemon, and the user-level %kiln, the "systemd of Urbit," needs a rethink to make upgrade logic state-triggered, not edge-triggered.

The console and the dojo need some work to make our user experience real. One way to view a high-usability command line is that a command, if you don't have the esoteric knowledge to fill it in by just typing the syntax, is essentially a form. This form needs to work both in the browser and the terminal, which isn't trivial. Browser and terminal both need a simple multiscreen navigation model, which we don't have yet.

There's also a bunch of Unix-level work to do, mainly in converting the old "single-player mode" console-bound Urbit into a real system service. Urbit works better as a daemon which runs one or more worker processes, and talks to client consoles over a Unix socket. A real architecture actually makes it practical for star owners to host the planets they issue, for example.

Finally, because we never forget the wisdom of the sages, our last act before asking people to use Urbit will be the long awaited performance crusade. The new event system now at least releases outgoing events before the input queue is clear! You never know what optimization will achieve before you make a serious effort, but I don't see any reason we can't carve out a new empire of speed in the savage, uncharted east.

We haven't touched the documentation since the first half of 2016. As always, maturity has to come before documentation. Premature documentation is almost as dangerous as premature optimization.

Notable contributions

Some notable contributions of late:

  • ~ponnys-podfer wrote a MUD
  • ~palfun-foslup created a surprisingly capable talkbot
  • ~master-morzod reworked the console in both the Hoon and C layers, adding color and cleaning up a bunch of stuff.

Anyone who contributes at this point deserves extreme praise for the ability to work in an immature, often broken system. We look forward to being ready for normal programmers who aren't intrepid jungle explorers.

Next crowdsale

We'll do another crowdsale in the spring. We understand that the last sale left a lot of people unable to participate — and our first priority is making sure those who are excited about Urbit can own some real estate. We'll announce our plans on how this might happen soon. If you're interested, make sure you're on our mailing list.

Contribution prizes

We're still thinking about the right way to award Urbit real estate to outside contributors. Especially after the crowdsale, it really has to feel 100% fair. But it can't be in any way, whether legally or emotionally, a form of compensation. We can't afford to kill the fun.

It doesn't feel right to do this at all when the documentation is completely inadequate, because it's not fair to people who don't have the time to reverse engineer. But we're thinking hard about this issue.

Future updates

We know you love these opinionated, literary state-of-the-urbit updates. We love writing them. But we obsess over them too long and they take too long to write. Urbit's ultimate success depends almost entirely on the code.

Going forward we're going to switch to commit-message style updates on a monthly basis, with longer updates arriving sporadically. Maker DAO does a great job of these shorter updates (here is an example).

If you're ever curious to check in or hear what's going on, don't hesitate to get in touch. You can always chat with us on urbit.org/stream, or by booting your own Urbit. Twitter or email (urbit@urbit.org) are also perfectly fine.

There's also our forum that anyone can post to. We've created this thread to discuss this post.