Module chunk 

ColumnChunk: differential’s Chunk over Column-shaped updates.

A chunk is a sorted, consolidated run of (D, T, R) updates in the flat columnar layout, in one of two homes:

• Resident: an Rc-shared Column on the heap. Fresh input, merge output, and small tails live here.
• Spilled: the serialized body in the process Pool, with the record count and the first and last data items resident. The pool owns residency from there, with slots under a memory budget and compression and device pageout under pressure, and a body that dies before pressure reaches it is freed without I/O.

Reads of a spilled body are copy-out and scoped to the call that needs them: the body is read into caller-owned memory and no reference into pool memory ever exists outside the pool. That contract is what lets the pool evict with no reader accounting at all.

Spilling happens in Chunk::settle, the trait’s designated commit point: chunks moved to settled output are handed to the pool when spilling is enabled (see set_compute_spill_enabled and set_storage_spill_enabled). The spill destination resolves per commit from three pieces of mutable state. A thread-local pool override, for tests and benches, wins outright. Otherwise the compute and storage gates, composed as an OR, route commits to the process pool installed by crate::pool_config, and with no pool installed chunks stay resident. A second thread-local holds the reusable scratch that call-scoped reads of spilled bodies copy into. Grading is by serialized bytes, the ship size Column already targets, rather than by the record-count TARGET, since record count does not bound bytes for variable-width data.

Chunks whose data is a (key, val) pair additionally implement UnloadChunk, the bulk-read capability: sorted probe keys in, matching updates appended to caller-owned staging, with locate answered from the resident fence metadata so a probe set faults only the chunk bodies it actually touches.

Structs

ChunkChunker

A chunker for arrange_core over ColumnChunks: sorts and consolidates raw input columns through a ColumnChunker and wraps its output chunks.

Lz4Codec

The chunk-side ExtentCodec: a little-endian u32 body-length prefix followed by one lz4 block, the framing lz4_flex::block::compress_prepend_size produces. Every chunk consumer passes LZ4_CODEC at insert; the pool itself has no codec opinion.

SpilledBody

A spilled chunk body: the serialized column in the pool, plus the resident metadata every Chunk must answer without fetching. That metadata is the record count and the first and last data items (the fence entries UnloadChunk::locate consults).

UnchunkBuilder

A batch builder over ColumnChunk input that delegates to a builder over Column input, loading each chunk’s body as it is pushed.

Enums

ColumnChunk

A sorted, consolidated run of (D, T, R) updates, resident or spilled.

Constants

COMMIT_BYTES 🔒

The serialized-byte size committed chunks aim for, matching the ship size of the columnar merge machinery.

READ_SCRATCH 🔒

Reusable staging for call-scoped reads of spilled bodies.

SCRATCH_RETAIN_WORDS 🔒

Scratch capacity retained across reads, in words. A read larger than this releases the buffer afterward, so a thread’s scratch does not ratchet to the largest body it ever carried (heap no pool gauge can see).

SPILL_MIN_BYTES 🔒

Bodies smaller than this stay resident: the pool’s smallest size class is 64 KiB, so spilling below it trades no meaningful memory for slot waste.

SPILL_OVERRIDE 🔒

A thread-scoped pool override, taking precedence over the global enable flag and pool. Lets tests and benches spill through a private pool without touching process-global state.

Statics

COMPUTE_SPILL_ENABLED 🔒

Compute’s leg of the process spill gate. See set_compute_spill_enabled.

LZ4_CODEC

The Lz4Codec instance chunk consumers pass to Pool::insert_with.

STORAGE_SPILL_ENABLED 🔒

Storage’s leg of the process spill gate. See set_storage_spill_enabled.

Functions

at_commit_size 🔒

Whether a column is big enough to commit on its own. A monotone threshold, so settle’s carry, which grows by whole chunks, cannot step over it.

borrow_words 🔒

Reconstructs the borrowed columnar view from serialized words, the same zero-copy decode Column::borrow performs on its Align variant.

copy_column 🔒

Copy a column into a fresh Typed column via bulk per-leaf extension.

extract_view_into 🔒

Append every update in view whose key matches a probe at or after *probe_index into staging, per the UnloadChunk consume-index protocol: probes strictly below the view’s last key are consumed, a probe equal to it is extracted but left for the next chunk.

rr 🔒

Narrow a columnar ref to a shorter lifetime, so refs from different borrows, such as a probe column and a chunk’s own columns, can be compared (the refs are lifetime-invariant).

set_compute_spill_enabled

Enable or disable chunk spilling on behalf of compute’s arrangement batchers.

set_spill_override

Set or unset the pool through which this thread’s chunk spills are routed, taking precedence over the gates and the process pool. None restores the global resolution.

set_storage_spill_enabled

Enable or disable chunk spilling on behalf of storage’s upsert dataflows.

spill_column 🔒

Serialize a column into a pool slot. The Align variant is already the serialized form and copies in directly. Other variants write their ContainerBytes encoding through a cursor over the slot memory. Sizing is exact, so a short or overlong write is a contract violation and panics.

spill_pool 🔒

The pool committed chunks spill to, if any.

to_typed 🔒

A column is Typed, or becomes one by copy. Merge and settle accumulate into Typed targets. Serialized variants arrive from spill reads and remote channels.

with_scratch 🔒

Run f with this thread’s read scratch, cleared of any previous use.