pub enum Plan<T = Timestamp> {
Show 13 variants Constant { rows: Result<Vec<(Row, T, Diff)>, EvalError>, }, Get { id: Id, keys: AvailableCollections, plan: GetPlan, }, Let { id: LocalId, value: Box<Plan<T>>, body: Box<Plan<T>>, }, LetRec { ids: Vec<LocalId>, values: Vec<Plan<T>>, body: Box<Plan<T>>, }, Mfp { input: Box<Plan<T>>, mfp: MapFilterProject, input_key_val: Option<(Vec<MirScalarExpr>, Option<Row>)>, }, FlatMap { input: Box<Plan<T>>, func: TableFunc, exprs: Vec<MirScalarExpr>, mfp: MapFilterProject, input_key: Option<Vec<MirScalarExpr>>, }, Join { inputs: Vec<Plan<T>>, plan: JoinPlan, }, Reduce { input: Box<Plan<T>>, key_val_plan: KeyValPlan, plan: ReducePlan, input_key: Option<Vec<MirScalarExpr>>, }, TopK { input: Box<Plan<T>>, top_k_plan: TopKPlan, }, Negate { input: Box<Plan<T>>, }, Threshold { input: Box<Plan<T>>, threshold_plan: ThresholdPlan, }, Union { inputs: Vec<Plan<T>>, }, ArrangeBy { input: Box<Plan<T>>, forms: AvailableCollections, input_key: Option<Vec<MirScalarExpr>>, input_mfp: MapFilterProject, },
}
Expand description

A rendering plan with as much conditional logic as possible removed.

Variants§

§

Constant

Fields

§rows: Result<Vec<(Row, T, Diff)>, EvalError>

Explicit update triples for the collection.

A collection containing a pre-determined collection.

§

Get

Fields

§id: Id

A global or local identifier naming the collection.

§keys: AvailableCollections

Arrangements that will be available.

The collection will also be loaded if available, which it will not be for imported data, but which it may be for locally defined data.

§plan: GetPlan

The actions to take when introducing the collection.

A reference to a bound collection.

This is commonly either an external reference to an existing source or maintained arrangement, or an internal reference to a Let identifier.

§

Let

Fields

§id: LocalId

The local identifier to be used, available to body as Id::Local(id).

§value: Box<Plan<T>>

The collection that should be bound to id.

§body: Box<Plan<T>>

The collection that results, which is allowed to contain Get stages that reference Id::Local(id).

Binds value to id, and then results in body with that binding.

This stage has the effect of sharing value across multiple possible uses in body, and is the only mechanism we have for sharing collection information across parts of a dataflow.

The binding is not available outside of body.

§

LetRec

Fields

§ids: Vec<LocalId>

The local identifiers to be used, available to body as Id::Local(id).

§values: Vec<Plan<T>>

The collection that should be bound to id.

§body: Box<Plan<T>>

The collection that results, which is allowed to contain Get stages that reference Id::Local(id).

Binds values to ids, evaluates them potentially recursively, and returns body.

All bindings are available to all bindings, and to body. The contents of each binding are initially empty, and then updated through a sequence of iterations in which each binding is updated in sequence, from the most recent values of all bindings.

§

Mfp

Fields

§input: Box<Plan<T>>

The input collection.

§mfp: MapFilterProject

Linear operator to apply to each record.

§input_key_val: Option<(Vec<MirScalarExpr>, Option<Row>)>

Whether the input is from an arrangement, and if so, whether we can seek to a specific value therein

Map, Filter, and Project operators.

This stage contains work that we would ideally like to fuse to other plan stages, but for practical reasons cannot. For example: reduce, threshold, and topk stages are not able to absorb this operator.

§

FlatMap

Fields

§input: Box<Plan<T>>

The input collection.

§func: TableFunc

The variable-record emitting function.

§exprs: Vec<MirScalarExpr>

Expressions that for each row prepare the arguments to func.

§mfp: MapFilterProject

Linear operator to apply to each record produced by func.

§input_key: Option<Vec<MirScalarExpr>>

The particular arrangement of the input we expect to use, if any

A variable number of output records for each input record.

This stage is a bit of a catch-all for logic that does not easily fit in map stages. This includes table valued functions, but also functions of multiple arguments, and functions that modify the sign of updates.

This stage allows a MapFilterProject operator to be fused to its output, and this can be very important as otherwise the output of func is just appended to the input record, for as many outputs as it has. This has the unpleasant default behavior of repeating potentially large records that are being unpacked, producing quadratic output in those cases. Instead, in these cases use a mfp member that projects away these large fields.

§

Join

Fields

§inputs: Vec<Plan<T>>

An ordered list of inputs that will be joined.

§plan: JoinPlan

Detailed information about the implementation of the join.

This includes information about the implementation strategy, but also any map, filter, project work that we might follow the join with, but potentially pushed down into the implementation of the join.

A multiway relational equijoin, with fused map, filter, and projection.

This stage performs a multiway join among inputs, using the equality constraints expressed in plan. The plan also describes the implementation strategy we will use, and any pushed down per-record work.

§

Reduce

Fields

§input: Box<Plan<T>>

The input collection.

§key_val_plan: KeyValPlan

A plan for changing input records into key, value pairs.

§plan: ReducePlan

A plan for performing the reduce.

The implementation of reduction has several different strategies based on the properties of the reduction, and the input itself. Please check out the documentation for this type for more detail.

§input_key: Option<Vec<MirScalarExpr>>

The particular arrangement of the input we expect to use, if any

Aggregation by key.

§

TopK

Fields

§input: Box<Plan<T>>

The input collection.

§top_k_plan: TopKPlan

A plan for performing the Top-K.

The implementation of reduction has several different strategies based on the properties of the reduction, and the input itself. Please check out the documentation for this type for more detail.

Key-based “Top K” operator, retaining the first K records in each group.

§

Negate

Fields

§input: Box<Plan<T>>

The input collection.

Inverts the sign of each update.

§

Threshold

Fields

§input: Box<Plan<T>>

The input collection.

§threshold_plan: ThresholdPlan

A plan for performing the threshold.

The implementation of reduction has several different strategies based on the properties of the reduction, and the input itself. Please check out the documentation for this type for more detail.

Filters records that accumulate negatively.

Although the operator suppresses updates, it is a stateful operator taking resources proportional to the number of records with non-zero accumulation.

§

Union

Fields

§inputs: Vec<Plan<T>>

The input collections

Adds the contents of the input collections.

Importantly, this is multiset union, so the multiplicities of records will add. This is in contrast to set union, where the multiplicities would be capped at one. A set union can be formed with Union followed by Reduce implementing the “distinct” operator.

§

ArrangeBy

Fields

§input: Box<Plan<T>>

The input collection.

§forms: AvailableCollections

A list of arrangement keys, and possibly a raw collection, that will be added to those of the input.

If any of these collection forms are already present in the input, they have no effect.

§input_key: Option<Vec<MirScalarExpr>>

The key that must be used to access the input.

§input_mfp: MapFilterProject

The MFP that must be applied to the input.

The input plan, but with additional arrangements.

This operator does not change the logical contents of input, but ensures that certain arrangements are available in the results. This operator can be important for e.g. the Join stage which benefits from multiple arrangements or to cap a Plan so that indexes can be exported.

Implementations§

Iterates through mutable references to child expressions.

Replace the plan with another one that has the collection in some additional forms.

This method converts a MirRelationExpr into a plan that can be directly rendered.

The rough structure is that we repeatedly extract map/filter/project operators from each expression we see, bundle them up as a MapFilterProject object, and then produce a plan for the combination of that with the next operator.

The method takes as an argument the existing arrangements for each bound identifier, which it will locally add to and remove from for Let bindings (by the end of the call it should contain the same bindings as when it started).

The result of the method is both a Plan, but also a list of arrangements that are certain to be produced, which can be relied on by the next steps in the plan. Each of the arrangement keys is associated with an MFP that must be applied if that arrangement is used, to back out the permutation associated with that arrangement. An empty list of arrangement keys indicates that only a Collection stream can be assumed to exist.

Convert the dataflow description into one that uses render plans.

Partitions the plan into parts many disjoint pieces.

This is used to partition Plan::Constant stages so that the work can be distributed across many workers.

Trait Implementations§

The type of Strategy used to generate values of type Self.
The type of parameters that arbitrary_with accepts for configuration of the generated Strategy. Parameters must implement Default.
Generates a Strategy for producing arbitrary values of type the implementing type (Self). The strategy is passed the arguments given in args. Read more
Generates a Strategy for producing arbitrary values of type the implementing type (Self). Read more
Returns a copy of the value. Read more
Performs copy-assignment from source. Read more
Appends global identifiers on which this plan depends to out.
Returns the global identifiers on which this plan depends. Read more
Formats the value using the given formatter. Read more
Deserialize this value from the given Serde deserializer. Read more
This method tests for self and other values to be equal, and is used by ==.
This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.
Convert a Self into a Proto value.
Consume and convert a Proto back into a Self value. Read more
Serialize this value into the given Serde serializer. Read more

Auto Trait Implementations§

Blanket Implementations§

Gets the TypeId of self. Read more
Immutably borrows from an owned value. Read more
Mutably borrows from an owned value. Read more
Compare self to key and return true if they are equal.

Returns the argument unchanged.

Converts to this type from a reference to the input type.
Attaches the provided Context to this type, returning a WithContext wrapper. Read more
Attaches the current Context to this type, returning a WithContext wrapper. Read more
Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
Instruments this type with the current Span, returning an Instrumented wrapper. Read more

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Wrap the input message T in a tonic::Request
The alignment of pointer.
The type for initializers.
Initializes a with the given initializer. Read more
Dereferences the given pointer. Read more
Mutably dereferences the given pointer. Read more
Drops the object pointed to by the given pointer. Read more
Upcasts this ProgressEventTimestamp to Any. Read more
Returns the name of the concrete type of this object. Read more
Should always be Self
The resulting type after obtaining ownership.
Creates owned data from borrowed data, usually by cloning. Read more
Uses borrowed data to replace owned data, usually by cloning. Read more
The type returned in the event of a conversion error.
Performs the conversion.
The type returned in the event of a conversion error.
Performs the conversion.
Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more