Skip to main content

Module consolidate

Module consolidate 

Source
Expand description

A ContainerBuilder that consolidates (D, T, R) updates and emits columnar containers.

Two-level buffering:

  1. AoS staging Vec<(D, T, R)> with a small cap so consolidate_updatesn log n cost stays bounded. Cancellations and same-key updates collapse here before reaching the column-shaped storage. A drain-multiple-of-half-cap trick keeps the leftover in staging, so cross-batch keys with the same (D, T) continue consolidating on the next sort.
  2. SoA accumulator: one sub-container per column (D::Container, T::Container, R::Container). Drains push in fixed-size chunks (DRAIN_CHUNK_ROWS) with three sequential per-column passes per chunk, so the inner pushes autovectorize. After each chunk, check the serialized size; once the accumulator reaches the flush threshold (90% of OUTPUT_TARGET_WORDS), serialize into an aligned Vec<u64> (no zero-fill — written by indexed::encode) and ship as Column::Align. Per-chunk granularity bounds overshoot to K * row_words. The trailing partial on finish ships as Column::Typed.

Generic over (D, T, R): Columnar via the columnar tuple decomposition <(D, T, R) as Columnar>::Container = (D::Container, T::Container, R::Container).

Structs§

ConsolidatingColumnBuilder
A container builder that consolidates (D, T, R) updates and emits Column<(D, T, R)>.