Module sources

Types and traits related to the introduction of changing collections into dataflow.

Re-exports

pub use crate::sources::envelope::SourceEnvelope;

pub use crate::sources::kafka::KafkaSourceConnection;

pub use crate::sources::load_generator::LoadGeneratorSourceConnection;

pub use crate::sources::mysql::MySqlSourceConnection;

pub use crate::sources::mysql::MySqlSourceExportDetails;

pub use crate::sources::postgres::PostgresSourceConnection;

pub use crate::sources::postgres::PostgresSourceExportDetails;

pub use crate::sources::sql_server::SqlServerSource;

pub use crate::sources::sql_server::SqlServerSourceExtras;

Modules

encoding

Types and traits related to the decoding of data for sources.

envelope

Types related to source envelopes

kafka

Types related kafka sources

load_generator

Types related to load generator sources

mysql

Types related to mysql sources

postgres

Types related to postgres sources

proto_compression

Nested message and enum types in ProtoCompression.

proto_ingestion_description

Nested message and enum types in ProtoIngestionDescription.

proto_source_connection

Nested message and enum types in ProtoSourceConnection.

proto_source_data

Nested message and enum types in ProtoSourceData.

proto_source_export_details

Nested message and enum types in ProtoSourceExportDetails.

proto_source_export_statement_details

Nested message and enum types in ProtoSourceExportStatementDetails.

proto_timeline

Nested message and enum types in ProtoTimeline.

sql_server

Types related to SQL Server sources

Structs

IngestionDescription

A description of a source ingestion

MzOffset

Universal language for describing message positions in Materialize, in a source independent way. Individual sources like Kafka or File sources should explicitly implement their own offset type that converts to/From MzOffsets. A 0-MzOffset denotes an empty stream.

ProtoCompression

ProtoIngestionDescription

ProtoMzOffset

ProtoSourceConnection

ProtoSourceData

ProtoSourceDesc

ProtoSourceExportDataConfig

ProtoSourceExportDetails

ProtoSourceExportStatementDetails

NOTE: this message is encoded and stored as part of a source export statement option (currently only CREATE SUBSOURCE statements) Be extra careful about changes, ensuring that all changes are backwards compatible

ProtoTimeline

SourceData

SourceDataColumnarDecoder

SourceDataColumnarEncoder

SourceDesc

An external source of updates for a relational collection.

SourceExport

SourceExportDataConfig

Defines the configuration for how to handle data that is exported for a given Source Export.

SourceReferenceResolver

Stores and resolves references to a &[T: ExternalCatalogReference].

Enums

Compression

ExternalReferenceResolutionError

GenericSourceConnection

SourceDataRowColumnarDecoder

A decoder for Rows within SourceData.

SourceDataRowColumnarEncoder

An encoder for Rows within SourceData.

SourceExportDetails

Details necessary for each source export to allow the source implementations to export data to the export’s collection.

SourceExportStatementDetails

Details necessary to store in the Details option of a source export statement (CREATE SUBSOURCE and CREATE TABLE .. FROM SOURCE statements), to generate the appropriate SourceExportDetails struct during planning. NOTE that this is serialized as proto to the catalog, so any changes here must be backwards compatible or will require a migration.

Timeline

The meaning of the timestamp number produced by data sources. This type is not concerned with the source of the timestamp (like if the data came from a Debezium consistency topic or a CDCv2 stream), instead only what the timestamp number means.

Traits

ExternalCatalogReference

Describes how external references should be organized in a multi-level hierarchy.

SourceConnection

A connection to an external system

SourceTimestamp

Functions

arb_source_data_for_relation_desc

Given a RelationDesc returns an arbitrary SourceData.