pub trait SourceRender {
type Time: SourceTimestamp;
const STATUS_NAMESPACE: StatusNamespace;
// Required method
fn render<G: Scope<Timestamp = Self::Time>>(
self,
scope: &mut G,
config: &RawSourceCreationConfig,
resume_uppers: impl Stream<Item = Antichain<Self::Time>> + 'static,
start_signal: impl Future<Output = ()> + 'static,
) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<Self::Time>>>, Vec<PressOnDropButton>);
}
Expand description
Describes a source that can render itself in a timely scope.
Required Associated Constants§
Required Associated Types§
type Time: SourceTimestamp
Required Methods§
Sourcefn render<G: Scope<Timestamp = Self::Time>>(
self,
scope: &mut G,
config: &RawSourceCreationConfig,
resume_uppers: impl Stream<Item = Antichain<Self::Time>> + 'static,
start_signal: impl Future<Output = ()> + 'static,
) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<Self::Time>>>, Vec<PressOnDropButton>)
fn render<G: Scope<Timestamp = Self::Time>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<Self::Time>> + 'static, start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<Self::Time>>>, Vec<PressOnDropButton>)
Renders the source in the provided timely scope.
The resume_uppers
stream can be used by the source to observe the overall progress of the
ingestion. When a frontier appears in this stream the source implementation can be certain
that future ingestion instances will request to read the external data only at times beyond
that frontier. Therefore, the source implementation can react to this stream by e.g
committing offsets upstream or advancing the LSN of a replication slot. It is safe to
ignore this argument.
Rendering a source is expected to return four things.
First, a source must produce a collection that is produced by the rendered dataflow and must contain definite[^1] data for all times beyond the resumption frontier.
Second, a source must produce a progress stream that will be used to drive reclocking. The frontier of this stream decides for which upstream offsets bindings can be minted.
Third, a source must produce a stream of health status updates.
Finally, the source is expected to return an opaque token that when dropped will cause the source to immediately drop all capabilities and advance its frontier to the empty antichain.
Dyn Compatibility§
This trait is not dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.
Implementations on Foreign Types§
Source§impl SourceRender for KafkaSourceConnection
impl SourceRender for KafkaSourceConnection
const STATUS_NAMESPACE: StatusNamespace = StatusNamespace::Kafka
type Time = Partitioned<RangeBound<i32>, MzOffset>
fn render<G: Scope<Timestamp = KafkaTimestamp>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<KafkaTimestamp>> + 'static, start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<KafkaTimestamp>>>, Vec<PressOnDropButton>)
Source§impl SourceRender for LoadGeneratorSourceConnection
impl SourceRender for LoadGeneratorSourceConnection
const STATUS_NAMESPACE: StatusNamespace = StatusNamespace::Generator
type Time = MzOffset
fn render<G: Scope<Timestamp = MzOffset>>( self, scope: &mut G, config: &RawSourceCreationConfig, committed_uppers: impl Stream<Item = Antichain<MzOffset>> + 'static, start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<MzOffset>>>, Vec<PressOnDropButton>)
Source§impl SourceRender for MySqlSourceConnection
impl SourceRender for MySqlSourceConnection
Source§fn render<G: Scope<Timestamp = GtidPartition>>(
self,
scope: &mut G,
config: &RawSourceCreationConfig,
resume_uppers: impl Stream<Item = Antichain<GtidPartition>> + 'static,
_start_signal: impl Future<Output = ()> + 'static,
) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<GtidPartition>>>, Vec<PressOnDropButton>)
fn render<G: Scope<Timestamp = GtidPartition>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<GtidPartition>> + 'static, _start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<GtidPartition>>>, Vec<PressOnDropButton>)
Render the ingestion dataflow. This function only connects things together and contains no actual processing logic.
const STATUS_NAMESPACE: StatusNamespace = StatusNamespace::MySql
type Time = Partitioned<Uuid, GtidState>
Source§impl SourceRender for PostgresSourceConnection
impl SourceRender for PostgresSourceConnection
Source§fn render<G: Scope<Timestamp = MzOffset>>(
self,
scope: &mut G,
config: &RawSourceCreationConfig,
resume_uppers: impl Stream<Item = Antichain<MzOffset>> + 'static,
_start_signal: impl Future<Output = ()> + 'static,
) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<MzOffset>>>, Vec<PressOnDropButton>)
fn render<G: Scope<Timestamp = MzOffset>>( self, scope: &mut G, config: &RawSourceCreationConfig, resume_uppers: impl Stream<Item = Antichain<MzOffset>> + 'static, _start_signal: impl Future<Output = ()> + 'static, ) -> (BTreeMap<GlobalId, StackedCollection<G, Result<SourceMessage, DataflowError>>>, Stream<G, Infallible>, Stream<G, HealthStatusMessage>, Stream<G, ProgressStatisticsUpdate>, Option<Stream<G, Probe<MzOffset>>>, Vec<PressOnDropButton>)
Render the ingestion dataflow. This function only connects things together and contains no actual processing logic.