Skip to main content

Module columnar

Module columnar 

Source
Expand description

Columnar dataflow edge support.

Defines CollectionEdge, a wrapper that lets dataflow edges between Plan nodes carry either row-based (VecCollection) or columnar (ColumnarCollection) batches of (D, T, R) updates.

§Migration model

The migration is consumer-first: every Plan-node consumer learns to accept both variants before any producer emits the columnar variant. Producers can then flip to columnar one at a time.

Within a Plan node, operators may freely materialize Vec collections; only the inter-node edge format is constrained. A decode from columnar to Vec at a consumer’s input is acceptable only when the consumer would have decoded Row to mz_repr::Datum anyway. Pure passthrough consumers (Negate, Union) round-trip the columnar variant without decoding.

Consumers that have not yet learned the columnar form fall back to CollectionEdge::into_vec, which decodes through the named ColumnarToVec operator. Repack seams therefore stay visible in dataflow introspection, so they can be found and retired.

Enums§

CollectionEdge
A dataflow edge carrying records as either a row-based VecCollection or a ColumnarCollection.

Functions§

columnar_negate
Negates the diff of every record in a ColumnarCollection.
columnar_to_vec
Decodes columnar batches into a row-based collection.
vec_to_columnar
Repacks a row-based collection into columnar batches.

Type Aliases§

ColumnarCollection
A columnar collection of (D, T, R) updates traveling on a compute dataflow edge.