mz_compute::logging

Function consolidate_and_pack

Source
pub(crate) fn consolidate_and_pack<G, B, CB, L, F>(
    input: &StreamCore<G, B::Input>,
    log: L,
    logic: F,
) -> StreamCore<G, CB::Container>
where G: Scope<Timestamp = Timestamp>, B: Batcher<Time = G::Timestamp> + 'static, B::Input: Container + Clone + 'static, B::Output: Container + Clone + 'static, CB: ContainerBuilder, L: Into<LogVariant>, F: for<'a> FnMut(<B::Output as Container>::ItemRef<'a>, &mut PermutedRowPacker, &mut Session<'_, Timestamp, CB, Counter<Timestamp, <CB as ContainerBuilder>::Container, Tee<Timestamp, <CB as ContainerBuilder>::Container>>>) + 'static,
Expand description

A single-purpose function to consolidate and pack updates for log collection.

The function first consolidates worker-local updates using the Pipeline pact, then converts the updates into (Row, Row) pairs using the provided logic function. It is crucial that the data is not exchanged between workers, as the consolidation would not function as desired otherwise.