Skip to main content

Module merge_batcher

Module merge_batcher 

Source
Expand description

Merge-batcher for Column chunks with per-chunk paging.

Forks the differential_dataflow merge-batcher framework so chains can hold PagedColumn entries — letting the ColumnPager page chunks out as they’re produced and fetch them back lazily during merge / extract.

Reuses the resident building blocks from super::batcher: the inherent Column::merge_from / Column::extract methods (per-chunk merge / split). Input consolidation happens upstream: the chunker (super::batcher::ColumnChunker) is supplied to the arrange operator separately, so this batcher receives already-consolidated Column chunks via PushInto.

Structs§

ColumnMergeBatcher
Drives the merge-batcher over Column chunks routed through a ColumnPager.
FetchIter
Streaming materializer over a chain of PagedColumn entries.

Functions§

extract_chain
Streaming extract: walks merged chunk-by-chunk via Column::extract, routing each filled keep/ship chunk through its sink after pageing. Mirrors the per-chunk ship-threshold yield already inside Column::extract.
merge_chains
Two-way merge driver. Reuses today’s per-chunk gallop / ship-threshold logic from Column::merge_from, but pulls heads from FetchIter and emits finished output chunks through sink after routing them through the pager exposed by FetchIter::pager.