Module mz_storage::source

Types related to the creation of dataflow raw sources.

Raw sources are streams (currently, Timely streams) of data directly produced by the upstream service. The main export of this module is create_raw_source, which turns RawSourceCreationConfigs, ExternalSourceConnections, and SourceReader implementations into the aforementioned streams.

The full source, which is the differential stream that represents the actual object created by a CREATE SOURCE statement, is created by composing create_raw_source with decoding, SourceEnvelope rendering, and more.

Modules

metrics

“Base” metrics used by all dataflow sources.

persist_source

A source that reads from an a persist shard.

proto_source_timestamp

Nested message and enum types in ProtoSourceTimestamp.

util

Structs

DecodeResult

The output of the decoding operator

DelimitedValueSource

A wrapper that converts a delimited source reader that only provides values into a key/value reader whose key is always None

KafkaMetadata

Kafka-specific information about the event

KafkaSourceReader

Contains all information necessary to ingest data from Kafka

KinesisSourceReader

Contains all information necessary to ingest data from Kinesis

PartitionMetrics

Partition-specific metrics, recorded to both Prometheus and a system table

PostgresSourceReader

Information required to sync data from Postgres

ProtoAssignedTimestamp

A timestamp assigned to a source message.

ProtoSourceTimestamp

Source-agnostic timestamp for source messages.

PubNubSourceReader

Information required to sync data from PubNub

RawSourceCreationConfig

Shared configuration information for all source types. This is used in the create_raw_source functions, which produce raw sources.

S3SourceReader

Information required to load data from S3

SourceMessage

Source-agnostic wrapper for messages. Each source must implement a conversion to Message.

SourceMetrics

Source-specific Prometheus metrics

SourceOutput

A record produced by a source

SourceReaderError

A structured error for SourceReader::get_next_message implementors. Also implements From<anyhow::Error> for convenience.

SourceToken

A SourceToken manages interest in a source.

Enums

NextMessage

SourceMessageType

A wrapper around SourceMessage that allows SourceReader’s to communicate if a message if the final message a specific offset

SourceStatus

The status of a source.

Traits

MaybeLength

Types that implement this trait expose a length function

SourceReader

This trait defines the interface between Materialize and external sources, and must be implemented for every new kind of source.

Functions

create_raw_source

Creates a raw source dataflow operator from a connection that has a corresponding SourceReader implentation. The type of ExternalSourceConnection determines the type of connection that should be created.

responsible_for

Returns true if the given source id/worker id is responsible for handling the given partition.