Expand description

Types and traits related to the introduction of changing collections into dataflow.

Modules

Types and traits related to the decoding of data for sources.

Nested message and enum types in ProtoCompression.

Nested message and enum types in ProtoDebeziumMode.

Nested message and enum types in ProtoDebeziumSourceProjection.

Nested message and enum types in ProtoExternalSourceConnection.

Nested message and enum types in ProtoIncludedColumnSource.

Nested message and enum types in ProtoIngestionDescription.

Nested message and enum types in ProtoKeyEnvelope.

Nested message and enum types in ProtoS3KeySource.

Nested message and enum types in ProtoSourceConnection.

Nested message and enum types in ProtoSourceData.

Nested message and enum types in ProtoSourceEnvelope.

Nested message and enum types in ProtoTimeline.

Nested message and enum types in ProtoUpsertStyle.

Structs

Enums

Ordered means we can trust Debezium high water marks

Debezium generates records that contain metadata about the upstream database. The structure of this metadata depends on the type of connection used. This struct records the relevant indices in the record, calculated during planning, so that the dataflow operator can unpack the structure and extract the relevant information.

Which piece of metadata a column corresponds to

Whether and how to include the decoded key of a stream in dataflows

A Source of Object Key names, the argument of the DISCOVER OBJECTS clause

A SourceConnection describes how data is produced for a source, be it from a local table, or some upstream service. It is the first step of rendering of a source, and describes only how to produce a stream of messages associated with MzOffset’s.

SourceEnvelopes describe how to turn a stream of messages from SourceConnections into a differential stream, that is, a stream of (data, time, diff) triples.

The meaning of the timestamp number produced by data sources. This type is not concerned with the source of the timestamp (like if the data came from a Debezium consistency topic or a CDCv2 stream), instead only what the timestamp number means.

UnplannedSourceEnvelope is a SourceEnvelope missing some information. This information is obtained in UnplannedSourceEnvelope::desc, where UnplannedSourceEnvelope::into_source_envelope creates a full SourceEnvelope

Functions

Legacy logic included something like an offset into almost data streams