pub struct DataflowDescription<P, S = (), T = Timestamp> {
    pub source_imports: BTreeMap<GlobalId, SourceInstanceDesc<S>>,
    pub index_imports: BTreeMap<GlobalId, (IndexDesc, RelationType)>,
    pub objects_to_build: Vec<BuildDesc<P>>,
    pub index_exports: BTreeMap<GlobalId, (IndexDesc, RelationType)>,
    pub sink_exports: BTreeMap<GlobalId, SinkDesc<S, T>>,
    pub as_of: Option<Antichain<T>>,
    pub debug_name: String,
    pub id: Uuid,
}
Expand description

A description of a dataflow to construct and results to surface.

Fields

source_imports: BTreeMap<GlobalId, SourceInstanceDesc<S>>

Sources instantiations made available to the dataflow.

index_imports: BTreeMap<GlobalId, (IndexDesc, RelationType)>

Indexes made available to the dataflow.

objects_to_build: Vec<BuildDesc<P>>

Views and indexes to be built and stored in the local context. Objects must be built in the specific order, as there may be dependencies of later objects on prior identifiers.

index_exports: BTreeMap<GlobalId, (IndexDesc, RelationType)>

Indexes to be made available to be shared with other dataflows (id of new index, description of index, relationtype of base source/view)

sink_exports: BTreeMap<GlobalId, SinkDesc<S, T>>

sinks to be created (id of new sink, description of sink)

as_of: Option<Antichain<T>>

An optional frontier to which inputs should be advanced.

If this is set, it should override the default setting determined by the upper bound of since frontiers contributing to the dataflow. It is an error for this to be set to a frontier not beyond that default.

debug_name: String

Human readable name

id: Uuid

Unique ID of the dataflow

Implementations

Creates a new dataflow description with a human-readable name.

Imports a previously exported index.

This method makes available an index previously exported as id, identified to the query by description (which names the view the index arranges, and the keys by which it is arranged).

Imports a source and makes it available as id.

Binds to id the relation expression plan.

Exports as id an index described by description.

Future uses of import_index in other dataflow descriptions may use id, as long as this dataflow has not been terminated in the meantime.

Exports as id a sink described by description.

Returns true iff id is already imported.

Assigns the as_of frontier to the supplied argument.

This method allows the dataflow to indicate a frontier up through which all times should be advanced. This can be done for at least two reasons: 1. correctness and 2. performance.

Correctness may require an as_of to ensure that historical detail is consolidated at representative times that do not present specific detail that is not specifically correct. For example, updates may be compacted to times that are no longer the source times, but instead some byproduct of when compaction was executed; we should not present those specific times as meaningfully different from other equivalent times.

Performance may benefit from an aggressive as_of as it reduces the number of distinct moments at which collections vary. Differential dataflow will refresh its outputs at each time its inputs change and to moderate that we can minimize the volume of distinct input times as much as possible.

Generally, one should consider setting as_of at least to the since frontiers of contributing data sources and as aggressively as the computation permits.

The number of columns associated with an identifier in the dataflow.

Identifiers of exported objects (indexes and sinks).

Returns the description of the object to build with the specified identifier.

Panics

Panics if id is not present in objects_to_build exactly once.

Computes the set of identifiers upon which the specified collection identifier depends.

id must specify a valid object in objects_to_build.

Like depends_on, but appends to an existing BTreeSet.

Determine a unique id for this dataflow based on the indexes it exports.

Determine if a dataflow description is compatible with this dataflow description.

Compatible dataflows have equal exports, imports, and objects to build. The as_of of the receiver has to be less equal the other as_of.

Trait Implementations

The type of Strategy used to generate values of type Self. Read more

The type of parameters that arbitrary_with accepts for configuration of the generated Strategy. Parameters must implement Default. Read more

Generates a Strategy for producing arbitrary values of type the implementing type (Self). The strategy is passed the arguments given in args. Read more

Generates a Strategy for producing arbitrary values of type the implementing type (Self). Read more

Returns a copy of the value. Read more

Performs copy-assignment from source. Read more

Formats the value using the given formatter. Read more

Deserialize this value from the given Serde deserializer. Read more

This method tests for self and other values to be equal, and is used by ==. Read more

This method tests for !=.

Convert a Self into a Proto value.

Consume and convert a Proto back into a Self value. Read more

Serialize this value into the given Serde serializer. Read more

Auto Trait Implementations

Blanket Implementations

Gets the TypeId of self. Read more

Immutably borrows from an owned value. Read more

Mutably borrows from an owned value. Read more

Compare self to key and return true if they are equal.

Returns the argument unchanged.

Attaches the provided Context to this type, returning a WithContext wrapper. Read more

Attaches the current Context to this type, returning a WithContext wrapper. Read more

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more

Instruments this type with the current Span, returning an Instrumented wrapper. Read more

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Wrap the input message T in a tonic::Request

Upcasts this ProgressEventTimestamp to Any. Read more

Returns the name of the concrete type of this object. Read more

Should always be Self

The resulting type after obtaining ownership.

Creates owned data from borrowed data, usually by cloning. Read more

🔬 This is a nightly-only experimental API. (toowned_clone_into)

Uses borrowed data to replace owned data, usually by cloning. Read more

The type returned in the event of a conversion error.

Performs the conversion.

The type returned in the event of a conversion error.

Performs the conversion.

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more