Module source

Types related to the creation of dataflow raw sources.

Raw sources are streams (currently, Timely streams) of data directly produced by the upstream service. The main export of this module is create_raw_source, which turns RawSourceCreationConfigs and SourceConnections implementations into the aforementioned streams.

The full source, which is the differential stream that represents the actual object created by a CREATE SOURCE statement, is created by composing create_raw_source with decoding, SourceEnvelope rendering, and more.

Modules

generator

kafka 🔒

mysql 🔒

Code to render the ingestion dataflow of a MySqlSourceConnection.

postgres 🔒

Code to render the ingestion dataflow of a PostgresSourceConnection. The dataflow consists of multiple operators in order to take advantage of all the available workers.

probe 🔒

Support for sending frontier probes to upstream systems.

reclock 🔒

source_reader_pipeline 🔒

Types related to the creation of dataflow raw sources.

sql_server 🔒

Code to render the ingestion dataflow of a SqlServerSource.

statistics 🔒

types

Types related to the source ingestion pipeline/framework.

Structs

KafkaSourceReader

Contains all information necessary to ingest data from Kafka

RawSourceCreationConfig

Shared configuration information for all source types. This is used in the create_raw_source functions, which produce raw sources.

SourceExportCreationConfig

Reduced version of RawSourceCreationConfig that is used when rendering each export.

Functions

create_raw_source

Creates a source dataflow operator graph from a source connection. The type of SourceConnection determines the type of connection that should be created.