Struct Chunk 

struct Chunk<D: Data> {
    paged: Mutex<Option<PagedColumn<(D, Timestamp, Diff)>>>,
    resident: OnceLock<Column<(D, Timestamp, Diff)>>,
    len: usize,
    first_time: Timestamp,
    last_time: Timestamp,
}

A non-empty chunk of updates, backed by a columnar region.

All updates in a chunk are sorted by (time, data) and consolidated.

Chunks are immutable once created. They are produced by ChunkBuilder, which mints a new chunk whenever its in-progress columnar container reaches a fixed serialized byte boundary (~2 MiB, matching the ship granularity used elsewhere in the codebase), so each chunk corresponds to a single, predictably sized allocation.

Fields

paged: Mutex<Option<PagedColumn<(D, Timestamp, Diff)>>>

The paged-out form, taken on first materialization.

A Mutex (not RefCell) keeps the chunk Sync: cursors hold chunks behind a shared Rc, and the iterator returned by CorrectionV2::updates_before borrows them across the persist writer’s await, so &Chunk must be Send. The lock is taken once, at materialization, and is otherwise uncontended (the sink runs single-threaded per worker).

resident: OnceLock<Column<(D, Timestamp, Diff)>>

The materialized form, populated lazily by Chunk::column on first access.

An OnceLock (not OnceCell) for the same Sync reason. Once set the slot is never cleared, so its address is stable and Chunk::index can hand out Ref<'_> borrows tied to &self. The allocation is freed when the chunk drops, which bounds resident memory to the chunks under an active merge front.

len: usize

Number of updates, cached so len and chain bookkeeping never page the chunk in.

first_time: Timestamp

Time of the first update, cached so boundary checks (split_at_time, can_accept) route a resting chunk without materializing it.

last_time: Timestamp

Time of the last update, cached likewise.

Implementations

impl<D: Data> Chunk<D>

fn from_column(data: Column<(D, Timestamp, Diff)>) -> Self

Page the given non-empty column out into a chunk.

Reads the cached metadata (length, boundary times) while the column is still resident, then hands it to the global column pager. The policy decides whether it actually spills; either way the chunk is born paged and materializes lazily on first read.

Panics

Panics if the column is empty. Chunks are non-empty by construction; ChunkBuilder only ever builds a chunk from a populated column.

fn column(&self) -> &Column<(D, Timestamp, Diff)>

Materialize the chunk’s column, paging it in on first access.

The returned reference is valid for as long as &self: the OnceLock slot is never cleared once populated, so its contents have a stable address.

fn len(&self) -> usize

Return the number of updates in the chunk.

fn index(&self, idx: usize) -> Ref<'_, (D, Timestamp, Diff)>

Return the update at the given index, paging the chunk in if necessary.

Panics

Panics if the given index is not populated.

fn first(&self) -> Ref<'_, (D, Timestamp, Diff)>

Return the first update in the chunk, paging the chunk in if necessary.

fn last(&self) -> Ref<'_, (D, Timestamp, Diff)>

Return the last update in the chunk, paging the chunk in if necessary.

fn first_time(&self) -> Timestamp

Return the time of the first update, without materializing the chunk.

fn last_time(&self) -> Timestamp

Return the time of the last update, without materializing the chunk.

fn find_time_greater_than(&self, time: Timestamp) -> Option<usize>

Return the index of the first update at a time greater than time, or None if no such update exists.

The early-out uses the cached last time, so a chunk whose updates are all at or before time is skipped without paging it in.

fn get_size(&self) -> SizeMetrics

Return the size of the chunk, for use in metrics.

Reports resident bytes only: a chunk still spilled (on swap or in a pager file) is not part of RSS and contributes nothing, matching the accounting in mz_timely_util::columnar::merge_batcher.

Trait Implementations

impl<D: Data> Debug for Chunk<D>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.