Struct dataflow::render::context::CollectionBundle [−][src]
pub struct CollectionBundle<S: Scope, V: Data, T: Lattice> where
T: Timestamp + Lattice,
S::Timestamp: Lattice + Refines<T>, {
pub collection: Option<(Collection<S, V, Diff>, Collection<S, DataflowError, Diff>)>,
pub arranged: BTreeMap<Vec<MirScalarExpr>, ArrangementFlavor<S, V, T>>,
}
Expand description
A bundle of the various ways a collection can be represented.
This type maintains the invariant that it does contain at least one valid source of data, either a collection or at least one arrangement.
Fields
collection: Option<(Collection<S, V, Diff>, Collection<S, DataflowError, Diff>)>
arranged: BTreeMap<Vec<MirScalarExpr>, ArrangementFlavor<S, V, T>>
Implementations
pub fn from_collections(
oks: Collection<S, V, Diff>,
errs: Collection<S, DataflowError, Diff>
) -> Self
pub fn from_collections(
oks: Collection<S, V, Diff>,
errs: Collection<S, DataflowError, Diff>
) -> Self
Construct a new collection bundle from update streams.
pub fn from_expressions(
exprs: Vec<MirScalarExpr>,
arrangements: ArrangementFlavor<S, V, T>
) -> Self
pub fn from_expressions(
exprs: Vec<MirScalarExpr>,
arrangements: ArrangementFlavor<S, V, T>
) -> Self
Inserts arrangements by the expressions on which they are keyed.
pub fn from_columns<I: IntoIterator<Item = usize>>(
columns: I,
arrangements: ArrangementFlavor<S, V, T>
) -> Self
pub fn from_columns<I: IntoIterator<Item = usize>>(
columns: I,
arrangements: ArrangementFlavor<S, V, T>
) -> Self
Inserts arrangements by the columns on which they are keyed.
pub fn as_specific_collection(
&self,
key: Option<&[MirScalarExpr]>
) -> (Collection<S, Row, Diff>, Collection<S, DataflowError, Diff>)
pub fn as_specific_collection(
&self,
key: Option<&[MirScalarExpr]>
) -> (Collection<S, Row, Diff>, Collection<S, DataflowError, Diff>)
Asserts that the arrangement for a specific key (or the raw collection for no key) exists, and returns the corresponding collection.
This returns the collection as-is, without doing any unthinning transformation. Therefore, it should be used when the appropriate transformation was planned as part of a following MFP.
pub fn flat_map<I, C, L>(
&self,
key_val: Option<(Vec<MirScalarExpr>, Option<Row>)>,
constructor: C
) -> (Stream<S, I::Item>, Collection<S, DataflowError, Diff>) where
I: IntoIterator,
I::Item: Data,
C: FnOnce() -> L,
L: for<'a, 'b> FnMut(&'a [&'b RefOrMut<'b, Row>], &'a S::Timestamp, &'a Diff) -> I + 'static,
pub fn flat_map<I, C, L>(
&self,
key_val: Option<(Vec<MirScalarExpr>, Option<Row>)>,
constructor: C
) -> (Stream<S, I::Item>, Collection<S, DataflowError, Diff>) where
I: IntoIterator,
I::Item: Data,
C: FnOnce() -> L,
L: for<'a, 'b> FnMut(&'a [&'b RefOrMut<'b, Row>], &'a S::Timestamp, &'a Diff) -> I + 'static,
Constructs and applies logic to elements of a collection and returns the results.
constructor
takes a permutation and produces the logic to apply on elements. The logic
conceptually receives (&Row, &Row)
pairs in the form of a slice. Only after borrowing
the elements and applying the permutation the datums will be in the expected order.
If key_val
is set, this is a promise that logic
will produce no results on
records for which the key does not evaluate to the value. This is used when we
have an arrangement by that key to leap directly to exactly those records.
It is important that logic
still guard against data that does not satisfy
this constraint, as this method does not statically know that it will have
that arrangement.
fn flat_map_core<Tr, I, L>(
trace: &Arranged<S, Tr>,
key: Option<Row>,
logic: L,
refuel: usize
) -> Stream<S, I::Item> where
Tr: TraceReader<Key = Row, Val = Row, Time = S::Timestamp, R = Diff> + Clone + 'static,
Tr::Batch: BatchReader<Row, Tr::Val, S::Timestamp, Diff> + 'static,
Tr::Cursor: Cursor<Row, Tr::Val, S::Timestamp, Diff> + 'static,
I: IntoIterator,
I::Item: Data,
L: for<'a, 'b> FnMut(RefOrMut<'b, Row>, RefOrMut<'b, Row>, &'a S::Timestamp, &'a Diff) -> I + 'static,
fn flat_map_core<Tr, I, L>(
trace: &Arranged<S, Tr>,
key: Option<Row>,
logic: L,
refuel: usize
) -> Stream<S, I::Item> where
Tr: TraceReader<Key = Row, Val = Row, Time = S::Timestamp, R = Diff> + Clone + 'static,
Tr::Batch: BatchReader<Row, Tr::Val, S::Timestamp, Diff> + 'static,
Tr::Cursor: Cursor<Row, Tr::Val, S::Timestamp, Diff> + 'static,
I: IntoIterator,
I::Item: Data,
L: for<'a, 'b> FnMut(RefOrMut<'b, Row>, RefOrMut<'b, Row>, &'a S::Timestamp, &'a Diff) -> I + 'static,
Factored out common logic for using literal keys in general traces.
This logic is sufficiently interesting that we want to write it only once, and thereby avoid any skew in the two uses of the logic.
The function presents the contents of the trace as (key, value, time, delta)
tuples,
where key and value are rows.
Look up an arrangement by the expressions that form the key.
The result may be None
if no such arrangement exists, or it may be one of many
“arrangement flavors” that represent the types of arranged data we might have.
pub fn as_collection_core(
&self,
mfp: MapFilterProject,
key_val: Option<(Vec<MirScalarExpr>, Option<Row>)>
) -> (Collection<S, Row, Diff>, Collection<S, DataflowError, Diff>)
pub fn as_collection_core(
&self,
mfp: MapFilterProject,
key_val: Option<(Vec<MirScalarExpr>, Option<Row>)>
) -> (Collection<S, Row, Diff>, Collection<S, DataflowError, Diff>)
Presents self
as a stream of updates, having been subjected to mfp
.
This operator is able to apply the logic of mfp
early, which can substantially
reduce the amount of data produced when mfp
is non-trivial.
The key_val
argument, when present, indicates that a specific arrangement should
be used, and if, in addition, the val
component is present,
that we can seek to the supplied row.
pub fn ensure_collections(
self,
collections: AvailableCollections,
input_key: Option<Vec<MirScalarExpr>>,
input_mfp: MapFilterProject
) -> Self
Trait Implementations
Auto Trait Implementations
impl<S, V, T> !RefUnwindSafe for CollectionBundle<S, V, T>
impl<S, V, T> !Send for CollectionBundle<S, V, T>
impl<S, V, T> !Sync for CollectionBundle<S, V, T>
impl<S, V, T> Unpin for CollectionBundle<S, V, T> where
S: Unpin,
impl<S, V, T> !UnwindSafe for CollectionBundle<S, V, T>
Blanket Implementations
Mutably borrows from an owned value. Read more
Attaches the provided Subscriber
to this type, returning a
WithDispatch
wrapper. Read more
Attaches the current default Subscriber
to this type, returning a
WithDispatch
wrapper. Read more