Struct mz_compute::render::context::CollectionBundle

pub struct CollectionBundle<S: Scope, T = Timestamp>
where
    T: Timestamp + Lattice + Columnation,
    S::Timestamp: Lattice + Refines<T> + Columnation,
{
    pub collection: Option<(Collection<S, Row, Diff>, Collection<S, DataflowError, Diff>)>,
    pub arranged: BTreeMap<Vec<MirScalarExpr>, ArrangementFlavor<S, T>>,
}

A bundle of the various ways a collection can be represented.

This type maintains the invariant that it does contain at least one valid source of data, either a collection or at least one arrangement.

Fields

collection: Option<(Collection<S, Row, Diff>, Collection<S, DataflowError, Diff>)>

arranged: BTreeMap<Vec<MirScalarExpr>, ArrangementFlavor<S, T>>

Implementations

impl<S: Scope, T> CollectionBundle<S, T>

where T: Timestamp + Lattice + Columnation, S::Timestamp: Lattice + Refines<T> + Columnation,

pub fn from_collections( oks: Collection<S, Row, Diff>, errs: Collection<S, DataflowError, Diff>, ) -> Self

Construct a new collection bundle from update streams.

pub fn from_expressions( exprs: Vec<MirScalarExpr>, arrangements: ArrangementFlavor<S, T>, ) -> Self

Inserts arrangements by the expressions on which they are keyed.

pub fn from_columns<I: IntoIterator<Item = usize>>( columns: I, arrangements: ArrangementFlavor<S, T>, ) -> Self

Inserts arrangements by the columns on which they are keyed.

pub fn scope(&self) -> S

The scope containing the collection bundle.

pub fn enter_region<'a>( &self, region: &Child<'a, S, S::Timestamp>, ) -> CollectionBundle<Child<'a, S, S::Timestamp>, T>

Brings the collection bundle into a region.

impl<'a, S: Scope, T> CollectionBundle<Child<'a, S, S::Timestamp>, T>

where T: Timestamp + Lattice + Columnation, S::Timestamp: Lattice + Refines<T> + Columnation,

pub fn leave_region(&self) -> CollectionBundle<S, T>

Extracts the collection bundle from a region.

impl<S: Scope, T> CollectionBundle<S, T>

where T: Timestamp + Lattice + Columnation, S::Timestamp: Lattice + Refines<T> + Columnation,

pub fn as_specific_collection( &self, key: Option<&[MirScalarExpr]>, ) -> (Collection<S, Row, Diff>, Collection<S, DataflowError, Diff>)

Asserts that the arrangement for a specific key (or the raw collection for no key) exists, and returns the corresponding collection.

This returns the collection as-is, without doing any unthinning transformation. Therefore, it should be used when the appropriate transformation was planned as part of a following MFP.

pub fn flat_map<D, I, C, L>( &self, key_val: Option<(Vec<MirScalarExpr>, Option<Row>)>, constructor: C, ) -> (Stream<S, I::Item>, Collection<S, DataflowError, Diff>)

where I: IntoIterator<Item = (D, S::Timestamp, Diff)>, D: Data, C: FnOnce() -> L, L: for<'a, 'b> FnMut(&'a mut DatumVecBorrow<'b>, &'a S::Timestamp, &'a Diff) -> I + 'static,

Constructs and applies logic to elements of a collection and returns the results.

constructor takes a permutation and produces the logic to apply on elements. The logic conceptually receives (&Row, &Row) pairs in the form of a slice. Only after borrowing the elements and applying the permutation the datums will be in the expected order.

If key_val is set, this is a promise that logic will produce no results on records for which the key does not evaluate to the value. This is used when we have an arrangement by that key to leap directly to exactly those records. It is important that logic still guard against data that does not satisfy this constraint, as this method does not statically know that it will have that arrangement.

fn flat_map_core<Tr, K, D, I, L>( trace: &Arranged<S, Tr>, key: Option<K>, logic: L, refuel: usize, ) -> Stream<S, I::Item>

where for<'a> Tr::Key<'a>: ToDatumIter + IntoOwned<'a, Owned = K>, for<'a> Tr::Val<'a>: ToDatumIter, Tr: TraceReader<Time = S::Timestamp, Diff = Diff> + Clone + 'static, K: PartialEq + 'static, I: IntoIterator<Item = (D, Tr::Time, Tr::Diff)>, D: Data, L: for<'a, 'b> FnMut(Tr::Key<'_>, Tr::Val<'_>, &'a S::Timestamp, &'a Diff) -> I + 'static,

Factored out common logic for using literal keys in general traces.

This logic is sufficiently interesting that we want to write it only once, and thereby avoid any skew in the two uses of the logic.

The function presents the contents of the trace as (key, value, time, delta) tuples, where key and value are potentially specialized, but convertible into rows.

pub fn arrangement( &self, key: &[MirScalarExpr], ) -> Option<ArrangementFlavor<S, T>>

Look up an arrangement by the expressions that form the key.

The result may be None if no such arrangement exists, or it may be one of many “arrangement flavors” that represent the types of arranged data we might have.

impl<S, T> CollectionBundle<S, T>

where T: Timestamp + Lattice + Columnation, S: Scope, S::Timestamp: Refines<T> + Lattice + Timestamp + RenderTimestamp,

pub fn as_collection_core( &self, mfp: MapFilterProject, key_val: Option<(Vec<MirScalarExpr>, Option<Row>)>, until: Antichain<Timestamp>, ) -> (Collection<S, Row, Diff>, Collection<S, DataflowError, Diff>)

Presents self as a stream of updates, having been subjected to mfp.

This operator is able to apply the logic of mfp early, which can substantially reduce the amount of data produced when mfp is non-trivial.

The key_val argument, when present, indicates that a specific arrangement should be used, and if, in addition, the val component is present, that we can seek to the supplied row.

pub fn ensure_collections( self, collections: AvailableCollections, input_key: Option<Vec<MirScalarExpr>>, input_mfp: MapFilterProject, until: Antichain<Timestamp>, ) -> Self

fn specialized_arrange( name: &String, oks: Collection<S, Row, i64>, key: &Vec<MirScalarExpr>, thinning: &Vec<usize>, ) -> (MzArrangement<S>, Collection<S, DataflowError, i64>)

Builds a specialized arrangement to provided types. The specialization for key and value types of the arrangement is based on the bit length derived from the corresponding type descriptions.

Trait Implementations

impl<S: Clone + Scope, T> Clone for CollectionBundle<S, T>

where T: Timestamp + Lattice + Columnation + Clone, S::Timestamp: Lattice + Refines<T> + Columnation,

fn clone(&self) -> CollectionBundle<S, T>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.