Module mz_dataflow_types::sources
source · [−]Expand description
Types and traits related to the introduction of changing collections into dataflow
.
Modules
Types and traits related to the decoding of data for sources.
Nested message and enum types in ProtoCompression
.
Nested message and enum types in ProtoDebeziumMode
.
Nested message and enum types in ProtoDebeziumSourceProjection
.
Nested message and enum types in ProtoIncludedColumnSource
.
Nested message and enum types in ProtoIngestionDescription
.
Nested message and enum types in ProtoKeyEnvelope
.
Nested message and enum types in ProtoS3KeySource
.
Nested message and enum types in ProtoSourceConnection
.
Nested message and enum types in ProtoSourceData
.
Nested message and enum types in ProtoSourceEnvelope
.
Nested message and enum types in ProtoTimeline
.
Nested message and enum types in ProtoUpsertStyle
.
Structs
A column that was created via an INCLUDE
expression
A description of a source ingestion
Universal language for describing message positions in Materialize, in a source independent way. Individual sources like Kafka or File sources should explicitly implement their own offset type that converts to/From MzOffsets. A 0-MzOffset denotes an empty stream.
An external source of updates for a relational collection.
Enums
Ordered means we can trust Debezium high water marks
Debezium generates records that contain metadata about the upstream database. The structure of this metadata depends on the type of connection used. This struct records the relevant indices in the record, calculated during planning, so that the dataflow operator can unpack the structure and extract the relevant information.
Which piece of metadata a column corresponds to
Whether and how to include the decoded key of a stream in dataflows
A Source of Object Key names, the argument of the DISCOVER OBJECTS
clause
SourceEnvelope
s describe how to turn a stream of messages from SourceDesc
s
into a differential stream, that is, a stream of (data, time, diff)
triples.
The meaning of the timestamp number produced by data sources. This type is not concerned with the source of the timestamp (like if the data came from a Debezium consistency topic or a CDCv2 stream), instead only what the timestamp number means.
UnplannedSourceEnvelope
is a SourceEnvelope
missing some information. This information
is obtained in UnplannedSourceEnvelope::desc
, where
UnplannedSourceEnvelope::into_source_envelope
creates a full SourceEnvelope
Functions
Legacy logic included something like an offset into almost data streams