Function dataflow::decode::render_decode

pub fn render_decode<G>(
    stream: &Stream<G, SourceOutput<(), MessagePayload>>, 
    value_encoding: DataEncoding, 
    debug_name: &str, 
    metadata_items: Vec<IncludedColumnSource>, 
    operators: &mut Option<LinearOperator>, 
    metrics: Metrics
) -> (Stream<G, DecodeResult>, Option<Box<dyn Any>>) where
    G: Scope<Timestamp = Timestamp>, 

Decode arbitrary chunks of bytes into rows.

This decode API is used for upstream connectors that don’t discover delimiters themselves; i.e., those (like CSV files) that need help from the decoding stage to discover where one record ends and another begins.

As such, the connectors simply present arbitrary chunks of bytes about which we can’t assume any alignment properties. The DataDecoder API accepts these, and returns None if it needs more bytes to discover the boundary between messages. In that case, this function remembers the already-seen bytes and waits for new ones before calling into the decoder again.

If the decoder does find a message, we verify (by asserting) that it consumed some bytes, to avoid the possibility of infinite loops.