Network Explorer: Data Pipeline & HTTP API

Worker(s): Christopher King Reward: 5 Stars WIP

Overview

network explorer screenshot

Goal

Build a public, open HTTP API for exploring metrics about the Urbit network.

Background

The project known as the Network Explorer has been in the works at Tlon for quite some time. The idea is to build a publicly-accessible tool (hosted on https://network.urbit.org) that provides key metrics and insights into the state of the Urbit network. A series of high-fidelity designs have been produced.

Scope

This bounty covers the creation of the:

The Network Explorer interface will be covered in a separate bounty.

Implementation

Data Model

Let's start with the data model. There are two sources of data that we need:

Most of the required information in the Network Explorer will come from Azimuth. It's important to know though that the source of PKI information will likely change in the near future in light of ongoing work to reduce the cost of bringing nodes onto the network.

The below entity-relationship diagram defines a data model in terms of a relational database:

erd diagram

Some notes about the above diagram:

Data Ingestion

Data from Azimuth and radar will need to be regularly extracted, parsed, and persisted to the local database for querying.

For Azimuth, using a tool like infura should be used.. Tlon currently uses infura, so it should be possible to use a free account for development and Tlon's pre-existing service in production.

The output of the radar script is saved to a JSON file that can be found here. This file is updated as new ping information is obtained, so data ingestion from this service should happen on some regular schedule.

API

The following are are necessary API endpoints for the Network Explorer UI to function properly. It's likely that some details have been missed, so apply your own judgement when designing the API.

This interface is meant to be agnostic to the API mechanism you choose, which could be either REST or GraphQL.

get-node

Produce the details for a single node.

Parameters

Returns

get-nodes

Produce a list of nodes, optionally with parameters. Absence of parameters queries all nodes.

Parameters

Returns An array of node in ascending order by point.

get-pki-events

Produce the stream of PKI events for a specific Urbit ID, or globally in descending order of time.

Parameters

Returns An array of pki_event in descending order of time.

get-activity

Produce the network activity of a node or nodes.

Parameters

If urbit_id is unspecified, implies all ships.

If since and until are unspecified, produces all activity for all time. If only since is provided, produces activity from since to the present. If only until is provided, produces activity for all of time up until until. If both are provided, queries activity within a range.

Returns Array of:

Technology Requirements/Guidelines

That which is not explicitly specified here is up to the choice of the implementer.

Requirements

Resources

Milestones

Data ETL

2 stars The database has been designed and data from both sources is ingested (extracted, transformed and loaded) on a regular schedule. Analysis can be performed at the database layer.

HTTP API

2 stars An HTTP API is written per the above specification and deployed to https://api.network.urbit.org.

UI Integration

1 stars The Network Explorer UI will be a separate proposal, but integration with an interface inevitably surfaces oversights in the original design. An additional star will be awarded for providing ongoing maintenance as-needed during front-end integration.