Module consolidate 

A ContainerBuilder that consolidates (D, T, R) updates and emits columnar containers.

Two-level buffering:

1. AoS staging Vec<(D, T, R)> with a small cap so consolidate_updates’ n log n cost stays bounded. Cancellations and same-key updates collapse here before reaching the column-shaped storage. A drain-multiple-of-half-cap trick keeps the leftover in staging, so cross-batch keys with the same (D, T) continue consolidating on the next sort.
2. SoA accumulator: one sub-container per column (D::Container, T::Container, R::Container). Drains push in fixed-size chunks (DRAIN_CHUNK_ROWS) with three sequential per-column passes per chunk, so the inner pushes autovectorize. After each chunk, check the serialized size; once the accumulator reaches the flush threshold (90% of OUTPUT_TARGET_WORDS), serialize into an aligned Vec<u64> (no zero-fill — written by indexed::encode) and ship as Column::Align. Per-chunk granularity bounds overshoot to K * row_words. The trailing partial on finish ships as Column::Typed.

Generic over (D, T, R): Columnar via the columnar tuple decomposition <(D, T, R) as Columnar>::Container = (D::Container, T::Container, R::Container).

Structs

ConsolidatingColumnBuilder

A container builder that consolidates (D, T, R) updates and emits Column<(D, T, R)>.

Constants

DRAIN_CHUNK_ROWS 🔒

Drain rows from staging in chunks of this size. Inside each chunk we do three sequential per-column passes — long enough for autovectorization (4–8 vector iterations on NEON / SVE2 128-bit / AVX2 / SVE 256-bit) — while the outer per-chunk size check bounds overshoot to K * row_words. With a 1 KiB row (128 words) and K=16, worst-case overshoot is 2 KiB, well under the 10% slop budget on a 2 MiB target.

FLUSH_THRESHOLD_WORDS 🔒

Flush when within 10% of OUTPUT_TARGET_WORDS — matches ColumnBuilder’s slop heuristic. Computed at compile time so the hot loop is a single cmp/jae instead of a per-row round-up + divide-by-10.

OUTPUT_TARGET_WORDS 🔒

Target serialized chunk size in u64 words (2 MiB). Bounded slop matters once we put these behind huge pages.

STAGING_BUFFER_BYTES 🔒

Per-buffer byte budget for the staging cap. Matches the 8 KiB basis DD’s ConsolidatingContainerBuilder uses via timely::container::buffer::default_capacity.

Functions

default_staging_cap 🔒

Default items per staging buffer: 2 * STAGING_BUFFER_BYTES / size_of::<(D, T, R)>(), matching DD’s ConsolidatingContainerBuilder. Small enough that O(n log n) sort stays cheap, large enough to amortize the sort + drain overhead across many pushes.