Function mz_compute::render::reduce::build_bucketed_stage
source · fn build_bucketed_stage<G>(
debug_name: &str,
input: Collection<G, ((Row, u64), Vec<Row>), Diff>,
aggrs: Vec<AggregateFunc>,
buckets: u64,
validating: bool
) -> (Collection<G, ((Row, u64), Vec<Row>), Diff>, Option<Collection<G, DataflowError, Diff>>)where
G: Scope,
G::Timestamp: Lattice,
Expand description
Build the dataflow for one stage of a reduction tree for multiple hierarchical aggregates.
buckets
indicates the number of buckets in this stage. We do some non
obvious trickery here to limit the memory usage per layer by internally
holding only the elements that were rejected by this stage. However, the
output collection maintains the ((key, bucket), (passing value)
for this
stage.
validating
indicates whether we want this stage to perform error detection
for invalid accumulations. Once a stage is clean of such errors, subsequent
stages can skip validation.