Enum mz_compute::render::continual_task::ContinualTaskSourceTransformer

pub(crate) enum ContinualTaskSourceTransformer {
    InsertsInput {
        source_id: GlobalId,
        with_snapshot: bool,
    },
    SelfReference {
        source_id: GlobalId,
    },
    NormalReference,
}

An encapsulation of the transformation logic necessary on data coming into a continual task.

NB: In continual task jargon, an “input” contains diffs and a “reference” is a normal source/collection.

Variants

InsertsInput

A collection containing, at each time T, exactly the inserts at time T in the transformed collection.

For example:

• Input: {} at 0, {1} at 1, {1} at 2, …
• Output: {} at 0, {1} at 1, {} at 2, …

We’ll presumably have the same for deletes eventually, but it’s not exposed in the SQL frontend yet.

Fields

source_id: GlobalId

with_snapshot: bool

SelfReference

A self-reference to the continual task’s output. This is essentially a timely feedback loop via the persist shard. See module rustdoc for how this works.

Fields

source_id: GlobalId

NormalReference

A normal collection (no-op transformation).

Implementations

impl ContinualTaskSourceTransformer

pub fn snapshot_mode(&self) -> SnapshotMode

The persist_source SnapshotMode to use when reading this source.

pub fn suppress_early_progress_as_of( &self, as_of: Antichain<Timestamp>, ) -> Antichain<Timestamp>

Returns the as_of to use with the suppress_early_progress operator for this source. See the module rustdoc for context.

pub fn transform<S: Scope<Timestamp = Timestamp>>( &self, oks: Collection<S, Row, Diff>, errs: Collection<S, DataflowError, Diff>, ) -> (Collection<S, Row, Diff>, Collection<S, DataflowError, Diff>, Collection<S, (), Diff>)

Performs the necessary transformation on the source collection.

Returns the transformed “oks” and “errs” collections. Also returns the appropriate ct_times collection used to inform the sink which times were changed in the inputs.