Expand description
A controller that provides an interface to the compute layer, and the storage layer below it.
The compute controller manages the creation, maintenance, and removal of compute instances. This involves ensuring the intended service state with the orchestrator, as well as maintaining a dedicated compute instance controller for each active compute instance.
For each compute instance, the compute controller curates the creation of indexes and sinks installed on the instance, the progress of readers through these collections, and their eventual dropping and resource reclamation.
The state maintained for a compute instance can be viewed as a partial map from GlobalId to
collection. It is an error to use an identifier before it has been “created” with
create_dataflow(). Once created, the controller holds a read capability for each output
collection of a dataflow, which is manipulated with set_read_policy(). Eventually, a
collection is dropped with drop_collections().
A dataflow can be in read-only or read-write mode. In read-only mode, the dataflow does not
modify any persistent state. Sending a allow_write message to the compute instance will
transition the dataflow to read-write mode, allowing it to write to persistent sinks.
Created dataflows will prevent the compaction of their inputs, including other compute collections but also collections managed by the storage layer. Each dataflow input is prevented from compacting beyond the allowed compaction of each of its outputs, ensuring that we can recover each dataflow to its current state in case of failure or other reconfiguration.
Re-exports§
pub use instance_client::InstanceClient;
Modules§
- error
- Errors returned by the compute controller.
- instance_
client - A client for communicating with a compute instance task.
Structs§
- Collection
Frontiers - The frontiers of a compute collection.
- Compute
Controller - A controller for the compute layer.
Enums§
- Compute
Controller Response - Responses from the compute controller.
- Peek
Notification - Notification and summary of a received and forwarded
crate::protocol::response::ComputeResponse::PeekResponse.