Enum dataflow::Command

pub enum Command {
    CreateDataflows(Vec<DataflowDescription<RenderPlan>>),
    DropSources(Vec<GlobalId>),
    DropSinks(Vec<GlobalId>),
    DropIndexes(Vec<GlobalId>),
    Peek {
        id: GlobalId,
        key: Option<Row>,
        conn_id: u32,
        timestamp: Timestamp,
        finishing: RowSetFinishing,
        map_filter_project: SafeMfpPlan,
    },
    CancelPeek {
        conn_id: u32,
    },
    Insert {
        id: GlobalId,
        updates: Vec<Update>,
    },
    AllowCompaction(Vec<(GlobalId, Antichain<Timestamp>)>),
    DurabilityFrontierUpdates(Vec<(GlobalId, Antichain<Timestamp>)>),
    AddSourceTimestamping {
        id: GlobalId,
        connector: SourceConnector,
        bindings: Vec<(PartitionId, Timestamp, MzOffset)>,
    },
    AdvanceSourceTimestamp {
        id: GlobalId,
        update: TimestampSourceUpdate,
    },
    DropSourceTimestamping {
        id: GlobalId,
    },
    AdvanceAllLocalInputs {
        advance_to: Timestamp,
    },
    EnableLogging(LoggingConfig),
    EnablePersistence(RuntimeClient),
}

Explicit instructions for timely dataflow workers.

Variants

CreateDataflows(Vec<DataflowDescription<RenderPlan>>)

Create a sequence of dataflows.

Each of the dataflows must contain as_of members that are valid for each of the referenced arrangements, meaning AllowCompaction should be held back to those values until the command. Subsequent commands may arbitrarily compact the arrangements; the dataflow runners are responsible for ensuring that they can correctly maintain the dataflows.

Tuple Fields of CreateDataflows

0: Vec<DataflowDescription<RenderPlan>>

DropSources(Vec<GlobalId>)

Drop the sources bound to these names.

Tuple Fields of DropSources

0: Vec<GlobalId>

DropSinks(Vec<GlobalId>)

Drop the sinks bound to these names.

Tuple Fields of DropSinks

0: Vec<GlobalId>

DropIndexes(Vec<GlobalId>)

Drop the indexes bound to these namees.

Tuple Fields of DropIndexes

0: Vec<GlobalId>

Peek

Peek at an arrangement.

This request elicits data from the worker, by naming an arrangement and some actions to apply to the results before returning them.

The timestamp member must be valid for the arrangement that is referenced by id. This means that AllowCompaction for this arrangement should not pass timestamp before this command. Subsequent commands may arbitrarily compact the arrangements; the dataflow runners are responsible for ensuring that they can correctly answer the Peek.

Fields of Peek

id: GlobalId

The identifier of the arrangement.

key: Option<Row>

An optional key that should be used for the arrangement.

conn_id: u32

The identifier of this peek request.

Used in responses and cancelation requests.

timestamp: Timestamp

The logical timestamp at which the arrangement is queried.

finishing: RowSetFinishing

Actions to apply to the result set before returning them.

map_filter_project: SafeMfpPlan

Linear operation to apply in-line on each result.

CancelPeek

Cancel the peek associated with the given conn_id.

Fields of CancelPeek

conn_id: u32

The identifier of the peek request to cancel.

Insert

Insert updates into the local input named id.

Fields of Insert

id: GlobalId

Identifier of the local input.

updates: Vec<Update>

A list of updates to be introduced to the input.

AllowCompaction(Vec<(GlobalId, Antichain<Timestamp>)>)

Enable compaction in views.

Each entry in the vector names a view and provides a frontier after which accumulations must be correct. The workers gain the liberty of compacting the corresponding maintained traces up through that frontier.

Tuple Fields of AllowCompaction

0: Vec<(GlobalId, Antichain<Timestamp>)>

DurabilityFrontierUpdates(Vec<(GlobalId, Antichain<Timestamp>)>)

Update durability information for sources.

Each entry names a source and provides a frontier before which the source can be exactly replayed across restarts (i.e. we can assign the same timestamps to all the same data)

Tuple Fields of DurabilityFrontierUpdates

0: Vec<(GlobalId, Antichain<Timestamp>)>

AddSourceTimestamping

Add a new source to be aware of for timestamping.

Fields of AddSourceTimestamping

id: GlobalId

The ID of the timestamped source

connector: SourceConnector

The connector for the timestamped source.

bindings: Vec<(PartitionId, Timestamp, MzOffset)>

Previously stored timestamp bindings.

AdvanceSourceTimestamp

Advance worker timestamp

Fields of AdvanceSourceTimestamp

id: GlobalId

The ID of the timestamped source

update: TimestampSourceUpdate

The associated update (RT or BYO)

DropSourceTimestamping

Drop all timestamping info for a source

Fields of DropSourceTimestamping

id: GlobalId

The ID id of the formerly timestamped source.

AdvanceAllLocalInputs

Advance all local inputs to the given timestamp.

Fields of AdvanceAllLocalInputs

advance_to: Timestamp

The timestamp to advance to.

EnableLogging(LoggingConfig)

Request that the logging sources in the contained configuration are installed.

Tuple Fields of EnableLogging

0: LoggingConfig

EnablePersistence(RuntimeClient)

Enable persistence.

Tuple Fields of EnablePersistence

0: RuntimeClient

Implementations

impl Command

pub fn partition_among(self, parts: usize) -> Vec<Self>

Partitions the command into parts many disjoint pieces.

This is used to subdivide commands that can be sharded across workers, for example the plan::Constant stages of dataflow plans, and the Command::Insert commands that may contain multiple updates.

Trait Implementations

impl Clone for Command

fn clone(&self) -> Command

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Command

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for Command

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl Serialize for Command

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.