Expand description

A controller that provides an interface to the compute layer, and the storage layer below it.

The compute controller manages the creation, maintenance, and removal of compute instances. This involves ensuring the intended service state with the orchestrator, as well as maintaining a dedicated compute instance controller for each active compute instance.

For each compute instance, the compute controller curates the creation of indexes and sinks installed on the instance, the progress of readers through these collections, and their eventual dropping and resource reclamation.

The state maintained for a compute instance can be viewed as a partial map from GlobalId to collection. It is an error to use an identifier before it has been “created” with create_dataflows(). Once created, the controller holds a read capability for each output collection of a dataflow, which is manipulated with set_read_policy(). Eventually, a collection is dropped with either drop_collections() or by allowing compaction to the empty frontier.

Created dataflows will prevent the compaction of their inputs, including other compute collections but also collections managed by the storage layer. Each dataflow input is prevented from compacting beyond the allowed compaction of each of its outputs, ensuring that we can recover each dataflow to its current state in case of failure or other reconfiguration.


replicated 🔒
A client backed by multiple replicas.


A wrapper around a ComputeController with a live connection to a storage controller.
A wrapper around Instance with a live storage controller.
State maintained about individual collections.
A controller for the compute layer.
An event describing a change in status of a compute process.
A read-only handle to a compute instance.
Resource allocations for a replica of a compute instance.
Replica configuration
Instance 🔒
The state we keep for a compute instance.


Responses from the compute controller.
Errors arising from compute commands.
Size or location of a replica
Logging configuration of a replica. Changing this type requires a catalog storage migration!
Describes the status of an orchestrated service.

Type Definitions

An abstraction allowing us to name different compute instances.