Function mz_storage::decode::render_decode
source · pub fn render_decode<G>(
input: &Collection<G, SourceOutput<(), ByteStream>, u32>,
value_encoding: DataEncoding,
debug_name: &str,
metadata_items: Vec<IncludedColumnSource>,
metrics: DecodeMetrics,
connection_context: &ConnectionContext
) -> (Collection<G, DecodeResult, Diff>, Option<Box<dyn Any + Send + Sync>>)where
G: Scope<Timestamp = Timestamp>,
Expand description
Decode arbitrary chunks of bytes into rows.
This decode API is used for upstream connections that don’t discover delimiters themselves; i.e., those (like CSV files) that need help from the decoding stage to discover where one record ends and another begins.
As such, the connections simply present arbitrary chunks of bytes about which
we can’t assume any alignment properties. The DataDecoder
API accepts these,
and returns None
if it needs more bytes to discover the boundary between messages.
In that case, this function remembers the already-seen bytes and waits for new ones
before calling into the decoder again.
If the decoder does find a message, we verify (by asserting) that it consumed some bytes, to avoid the possibility of infinite loops.