Struct mz_compute::compute_state::ComputeState

pub struct ComputeState {

    pub collections: BTreeMap<GlobalId, CollectionState>,
    pub dropped_collections: Vec<GlobalId>,
    pub traces: TraceManager,
    pub subscribe_response_buffer: Rc<RefCell<Vec<(GlobalId, SubscribeResponse)>>>,
    pub copy_to_response_buffer: Rc<RefCell<Vec<(GlobalId, CopyToResponse)>>>,
    pub pending_peeks: BTreeMap<Uuid, PendingPeek>,
    pub compute_logger: Option<Logger<ComputeEvent, WorkerIdentifier>>,
    pub persist_clients: Arc<PersistClientCache>,
    pub command_history: ComputeCommandHistory<UIntGauge>,
    pub linear_join_spec: LinearJoinSpec,
    pub metrics: ComputeMetrics,
    pub context: ComputeInstanceContext,
    pub worker_config: ConfigSet,
    pub hydration_rx: Receiver<HydrationEvent>,
    pub hydration_tx: Sender<HydrationEvent>,
    /* private fields */
}

Worker-local state that is maintained across dataflows.

This state is restricted to the COMPUTE state, the deterministic, idempotent work done between data ingress and egress.

Fields

collections: BTreeMap<GlobalId, CollectionState>

State kept for each installed compute collection.

Each collection has exactly one frontier. How the frontier is communicated depends on the collection type:

• Frontiers of indexes are equal to the frontier of their corresponding traces in the TraceManager.
• Persist sinks store their current frontier in CollectionState::sink_write_frontier.
• Subscribes report their frontiers through the subscribe_response_buffer.

dropped_collections: Vec<GlobalId>

Collections that were recently dropped and whose removal needs to be reported.

traces: TraceManager

The traces available for sharing across dataflows.

subscribe_response_buffer: Rc<RefCell<Vec<(GlobalId, SubscribeResponse)>>>

Shared buffer with SUBSCRIBE operator instances by which they can respond.

The entries are pairs of sink identifier (to identify the subscribe instance) and the response itself.

copy_to_response_buffer: Rc<RefCell<Vec<(GlobalId, CopyToResponse)>>>

Shared buffer with S3 oneshot operator instances by which they can respond.

The entries are pairs of sink identifier (to identify the s3 oneshot instance) and the response itself.

pending_peeks: BTreeMap<Uuid, PendingPeek>

Peek commands that are awaiting fulfillment.

compute_logger: Option<Logger<ComputeEvent, WorkerIdentifier>>

The logger, from Timely’s logging framework, if logs are enabled.

persist_clients: Arc<PersistClientCache>

A process-global cache of (blob_uri, consensus_uri) -> PersistClient. This is intentionally shared between workers.

command_history: ComputeCommandHistory<UIntGauge>

History of commands received by this workers and all its peers.

linear_join_spec: LinearJoinSpec

Specification for rendering linear joins.

metrics: ComputeMetrics

Metrics for this replica.

context: ComputeInstanceContext

Other configuration for compute

worker_config: ConfigSet

Per-worker dynamic configuration.

This is separate from the process-global ConfigSet and contains config options that need to be applied consistently with compute command order.

For example, for options that influence dataflow rendering it is important that all workers render the same dataflow with the same options. If these options were stored in a global ConfigSet, we couldn’t guarantee that all workers observe changes to them at the same point in the stream of compute commands. Storing per-worker configuration ensures that because each worker’s configuration is only updated once that worker observes the respective UpdateConfiguration command.

hydration_rx: Receiver<HydrationEvent>

Receiver of operator hydration events.

hydration_tx: Sender<HydrationEvent>

Transmitter of operator hydration events.

Copies of this sender are passed to the hydration logging operators.

Implementations

impl ComputeState

pub fn new( worker_id: usize, persist_clients: Arc<PersistClientCache>, metrics: ComputeMetrics, tracing_handle: Arc<TracingHandle>, context: ComputeInstanceContext ) -> Self

Construct a new ComputeState.

pub fn collection_exists(&self, id: GlobalId) -> bool

Return whether a collection with the given ID exists.

pub fn expect_collection(&self, id: GlobalId) -> &CollectionState

Return a reference to the identified collection.

Panics if the collection doesn’t exist.

pub fn expect_collection_mut(&mut self, id: GlobalId) -> &mut CollectionState

Return a mutable reference to the identified collection.

Panics if the collection doesn’t exist.

pub fn input_probe_for( &mut self, input_id: GlobalId, collection_ids: impl Iterator<Item = GlobalId> ) -> Handle<Timestamp>

Construct a new frontier probe for the given input and add it to the state of the given collections.

The caller is responsible for attaching the returned probe handle to the respective dataflow input stream.

pub fn dataflow_max_inflight_bytes(&self) -> Option<usize>

Returns the cc or non-cc version of “dataflow_max_inflight_bytes”, as appropriate to this replica.