Trait SourceRender

pub trait SourceRender {
    type Time: SourceTimestamp;

    const STATUS_NAMESPACE: StatusNamespace;

    // Required method
    fn render<G: Scope<Timestamp = Self::Time>>(
        self,
        scope: &mut G,
        config: &RawSourceCreationConfig,
        resume_uppers: impl Stream<Item = Antichain<Self::Time>> + 'static,
        start_signal: impl Future<Output = ()> + 'static,
    ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<Self::Time>>>, Vec<PressOnDropButton>);
}

Describes a source that can render itself in a timely scope.

Required Associated Constants

const STATUS_NAMESPACE: StatusNamespace

Required Associated Types

type Time: SourceTimestamp

Required Methods

fn render<G: Scope<Timestamp = Self::Time>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<Self::Time>> + 'static, start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<Self::Time>>>, Vec<PressOnDropButton>)

Renders the source in the provided timely scope.

The resume_uppers stream can be used by the source to observe the overall progress of the ingestion. When a frontier appears in this stream the source implementation can be certain that future ingestion instances will request to read the external data only at times beyond that frontier. Therefore, the source implementation can react to this stream by e.g committing offsets upstream or advancing the LSN of a replication slot. It is safe to ignore this argument.

Rendering a source is expected to return four things.

First, a source must produce a collection that is produced by the rendered dataflow and must contain definite[^1] data for all times beyond the resumption frontier.

Second, a source must produce a progress stream that will be used to drive reclocking. The frontier of this stream decides for which upstream offsets bindings can be minted.

Third, a source must produce a stream of health status updates.

Finally, the source is expected to return an opaque token that when dropped will cause the source to immediately drop all capabilities and advance its frontier to the empty antichain.

[^1] https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/20210831_correctness.md#describing-definite-data

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementations on Foreign Types

impl SourceRender for KafkaSourceConnection

const STATUS_NAMESPACE: StatusNamespace = StatusNamespace::Kafka

type Time = Partitioned<RangeBound<i32>, MzOffset>

fn render<G: Scope<Timestamp = KafkaTimestamp>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<KafkaTimestamp>> + 'static, start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<KafkaTimestamp>>>, Vec<PressOnDropButton>)

impl SourceRender for LoadGeneratorSourceConnection

const STATUS_NAMESPACE: StatusNamespace = StatusNamespace::Generator

type Time = MzOffset

fn render<G: Scope<Timestamp = MzOffset>>( self, scope: &mut G, config: &RawSourceCreationConfig, committed_uppers: impl Stream<Item = Antichain<MzOffset>> + 'static, start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<MzOffset>>>, Vec<PressOnDropButton>)

impl SourceRender for MySqlSourceConnection

fn render<G: Scope<Timestamp = GtidPartition>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<GtidPartition>> + 'static, _start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<GtidPartition>>>, Vec<PressOnDropButton>)

Render the ingestion dataflow. This function only connects things together and contains no actual processing logic.

const STATUS_NAMESPACE: StatusNamespace = StatusNamespace::MySql

type Time = Partitioned<Uuid, GtidState>

impl SourceRender for PostgresSourceConnection

fn render<G: Scope<Timestamp = MzOffset>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<MzOffset>> + 'static, _start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<MzOffset>>>, Vec<PressOnDropButton>)

Render the ingestion dataflow. This function only connects things together and contains no actual processing logic.

const STATUS_NAMESPACE: StatusNamespace = StatusNamespace::Postgres

type Time = MzOffset

Implementors