Function dataflow::source::create_source [−][src]
pub(crate) fn create_source<G, S: 'static>(
config: SourceConfig<'_, G>,
source_connector: &ExternalSourceConnector,
persist_config: Option<PersistentTimestampBindingsConfig<SourceTimestamp, AssignedTimestamp>>
) -> ((Stream<G, SourceOutput<S::Key, S::Value>>, Stream<G, ((SourceTimestamp, AssignedTimestamp), Timestamp, Diff)>, Stream<G, SourceError>), Option<SourceToken>) where
G: Scope<Timestamp = Timestamp>,
S: SourceReader,
Expand description
Creates a source dataflow operator. The type of ExternalSourceConnector determines the type of source that should be created
If persist_config
is Some
, this will emit and persist a stream of timestamp bindings and
use the persisted bindings on startup to seed initial source offsets and timestamp bindings.
The returned Stream
of persisted timestamp bindings can be used to track the persistence
frontier and should be used to seal up the backing collection to that frontier. This function
does not do any sealing and it is the responsibility of the caller to eventually do that, for
example using seal
.