Module mz_storage::source
source ยท Expand description
Types related to the creation of dataflow raw sources.
Raw sources are streams (currently, Timely streams) of data directly
produced by the upstream service. The main export of this module is
create_raw_source
, which turns
RawSourceCreationConfigs
and
SourceConnections
implementations into the aforementioned streams.
The full source, which is the differential stream that represents the
actual object created by a CREATE SOURCE
statement, is created by
composing create_raw_source
with decoding,
SourceEnvelope
rendering, and
more.
Modulesยง
- kafka ๐
- mysql ๐Code to render the ingestion dataflow of a
MySqlSourceConnection
. - postgres ๐Code to render the ingestion dataflow of a
PostgresSourceConnection
. The dataflow consists of multiple operators in order to take advantage of all the available workers. - reclock ๐Operators that transform collections that evolve with some timestamp
FromTime
into a collections that evolve with some other timestamp `IntoTime. - Types related to the creation of dataflow raw sources.
- statistics ๐
- Types related to the source ingestion pipeline/framework.
Structsยง
- Contains all information necessary to ingest data from Kafka
- Shared configuration information for all source types. This is used in the
create_raw_source
functions, which produce raw sources.
Functionsยง
- Creates a source dataflow operator graph from a source connection. The type of SourceConnection determines the type of connection that should be created.