Struct dataflow::server::Worker

struct Worker<'w, A> where
    A: Allocate,  {

    timely_worker: &'w mut TimelyWorker<A>,
    compute_state: ComputeState,
    storage_state: StorageState,
    materialized_logger: Option<Logger>,
    command_rx: Receiver<Command>,
    pending_peeks: Vec<PendingPeek>,
    response_tx: UnboundedSender<Response>,
    reported_frontiers: HashMap<GlobalId, Antichain<Timestamp>>,
    reported_bindings_frontiers: HashMap<GlobalId, Antichain<Timestamp>>,
    last_bindings_feedback: Instant,
    metrics: WorkerMetrics,
    now: NowFn,
    dataflow_source_metrics: SourceBaseMetrics,
    dataflow_sink_metrics: SinkBaseMetrics,
}

State maintained for each worker thread.

Much of this state can be viewed as local variables for the worker thread, holding state that persists across function calls.

Fields

timely_worker: &'w mut TimelyWorker<A>

The underlying Timely worker.

compute_state: ComputeState

The state associated with rendering dataflows.

storage_state: StorageState

The state associated with collection ingress and egress.

materialized_logger: Option<Logger>

The logger, from Timely’s logging framework, if logs are enabled.

command_rx: Receiver<Command>

The channel from which commands are drawn.

pending_peeks: Vec<PendingPeek>

Peek commands that are awaiting fulfillment.

response_tx: UnboundedSender<Response>

The channel over which frontier information is reported.

reported_frontiers: HashMap<GlobalId, Antichain<Timestamp>>

Tracks the frontier information that has been sent over response_tx.

reported_bindings_frontiers: HashMap<GlobalId, Antichain<Timestamp>>

Tracks the timestamp binding durability information that has been sent over response_tx.

last_bindings_feedback: Instant

Tracks the last time we sent binding durability info over response_tx.

metrics: WorkerMetrics

Metrics bundle.

now: NowFn

dataflow_source_metrics: SourceBaseMetrics

Metrics for the source-specific side of dataflows.

dataflow_sink_metrics: SinkBaseMetrics

Implementations

impl<'w, A> Worker<'w, A> where
    A: Allocate + 'w, 

fn initialize_logging(&mut self, logging: &LoggingConfig)

Initializes timely dataflow logging and publishes as a view.

fn shutdown_logging(&mut self)

Disables timely dataflow logging.

This does not unpublish views and is only useful to terminate logging streams to ensure that materialized can terminate cleanly.

fn run(&mut self)

Draws from dataflow_command_receiver until shutdown.

fn report_compute_frontiers(&mut self)

Send progress information to the coordinator.

fn report_timestamp_bindings(&mut self)

Send information about new timestamp bindings created by dataflow workers back to the coordinator.

fn update_rt_timestamps(&self)

Instruct all real-time sources managed by the worker to close their current timestamp and move to the next wall clock time.

Needs to be called periodically (ideally once per “timestamp_frequency” in order for real time sources to make progress.

fn handle_command(&mut self, cmd: Command)

fn handle_compute_command(&mut self, cmd: ComputeCommand)

fn handle_storage_command(&mut self, cmd: StorageCommand)

fn process_peeks(&mut self)

Scan pending peeks and attempt to retire each.

fn send_peek_response(&mut self, peek: PendingPeek, response: PeekResponse)

Sends a response for this peek’s resolution to the coordinator.

Note that this function takes ownership of the PendingPeek, which is meant to prevent multiple responses to the same peek.

fn process_tails(&mut self)

Scan the shared tail response buffer, and forward results along.

fn send_compute_response(&self, response: ComputeResponse)

Send a response to the coordinator.

fn send_storage_response(&self, response: StorageResponse)

Send a response to the coordinator.