mz_storage::source::types

Trait SourceRender

Source
pub trait SourceRender {
    type Time: SourceTimestamp;

    const STATUS_NAMESPACE: StatusNamespace;

    // Required method
    fn render<G: Scope<Timestamp = Self::Time>>(
        self,
        scope: &mut G,
        config: &RawSourceCreationConfig,
        resume_uppers: impl Stream<Item = Antichain<Self::Time>> + 'static,
        start_signal: impl Future<Output = ()> + 'static,
    ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<Self::Time>>>, Vec<PressOnDropButton>);
}
Expand description

Describes a source that can render itself in a timely scope.

Required Associated Constants§

Required Associated Types§

Required Methods§

Source

fn render<G: Scope<Timestamp = Self::Time>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<Self::Time>> + 'static, start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<Self::Time>>>, Vec<PressOnDropButton>)

Renders the source in the provided timely scope.

The resume_uppers stream can be used by the source to observe the overall progress of the ingestion. When a frontier appears in this stream the source implementation can be certain that future ingestion instances will request to read the external data only at times beyond that frontier. Therefore, the source implementation can react to this stream by e.g committing offsets upstream or advancing the LSN of a replication slot. It is safe to ignore this argument.

Rendering a source is expected to return four things.

First, a source must produce a collection that is produced by the rendered dataflow and must contain definite[^1] data for all times beyond the resumption frontier.

Second, a source must produce a progress stream that will be used to drive reclocking. The frontier of this stream decides for which upstream offsets bindings can be minted.

Third, a source must produce a stream of health status updates.

Finally, the source is expected to return an opaque token that when dropped will cause the source to immediately drop all capabilities and advance its frontier to the empty antichain.

[^1] https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/20210831_correctness.md#describing-definite-data

Dyn Compatibility§

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementations on Foreign Types§

Source§

impl SourceRender for KafkaSourceConnection

Source§

impl SourceRender for LoadGeneratorSourceConnection

Source§

const STATUS_NAMESPACE: StatusNamespace = StatusNamespace::Generator

Source§

type Time = MzOffset

Source§

fn render<G: Scope<Timestamp = MzOffset>>( self, scope: &mut G, config: &RawSourceCreationConfig, committed_uppers: impl Stream<Item = Antichain<MzOffset>> + 'static, start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<MzOffset>>>, Vec<PressOnDropButton>)

Source§

impl SourceRender for MySqlSourceConnection

Source§

fn render<G: Scope<Timestamp = GtidPartition>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<GtidPartition>> + 'static, _start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<GtidPartition>>>, Vec<PressOnDropButton>)

Render the ingestion dataflow. This function only connects things together and contains no actual processing logic.

Source§

const STATUS_NAMESPACE: StatusNamespace = StatusNamespace::MySql

Source§

type Time = Partitioned<Uuid, GtidState>

Source§

impl SourceRender for PostgresSourceConnection

Source§

fn render<G: Scope<Timestamp = MzOffset>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<MzOffset>> + 'static, _start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<MzOffset>>>, Vec<PressOnDropButton>)

Render the ingestion dataflow. This function only connects things together and contains no actual processing logic.

Source§

const STATUS_NAMESPACE: StatusNamespace = StatusNamespace::Postgres

Source§

type Time = MzOffset

Implementors§