pub fn render_decode<G>(
    input: &Collection<G, SourceOutput<(), ByteStream>, u32>,
    value_encoding: DataEncoding,
    debug_name: &str,
    metadata_items: Vec<IncludedColumnSource>,
    metrics: DecodeMetrics,
    connection_context: &ConnectionContext
) -> (Collection<G, DecodeResult, Diff>, Option<Box<dyn Any + Send + Sync>>)where
    G: Scope<Timestamp = Timestamp>,
Expand description

Decode arbitrary chunks of bytes into rows.

This decode API is used for upstream connections that don’t discover delimiters themselves; i.e., those (like CSV files) that need help from the decoding stage to discover where one record ends and another begins.

As such, the connections simply present arbitrary chunks of bytes about which we can’t assume any alignment properties. The DataDecoder API accepts these, and returns None if it needs more bytes to discover the boundary between messages. In that case, this function remembers the already-seen bytes and waits for new ones before calling into the decoder again.

If the decoder does find a message, we verify (by asserting) that it consumed some bytes, to avoid the possibility of infinite loops.