Expand description

Types related to the creation of dataflow raw sources.

Raw sources are streams (currently, Timely streams) of data directly produced by the upstream service. The main export of this module is create_raw_source, which turns RawSourceCreationConfigs, ExternalSourceConnections, and SourceReader implementations into the aforementioned streams.

The full source, which is the differential stream that represents the actual object created by a CREATE SOURCE statement, is created by composing create_raw_source with decoding, SourceEnvelope rendering, and more.


“Base” metrics used by all dataflow sources.

A source that reads from an a persist shard.

Nested message and enum types in ProtoSourceTimestamp.


The output of the decoding operator

A wrapper that converts a delimited source reader that only provides values into a key/value reader whose key is always None

Kafka-specific information about the event

Contains all information necessary to ingest data from Kafka

Contains all information necessary to ingest data from Kinesis

Partition-specific metrics, recorded to both Prometheus and a system table

Information required to sync data from Postgres

A timestamp assigned to a source message.

Source-agnostic timestamp for source messages.

Information required to sync data from PubNub

Shared configuration information for all source types. This is used in the create_raw_source functions, which produce raw sources.

Information required to load data from S3

Source-agnostic wrapper for messages. Each source must implement a conversion to Message.

Source-specific Prometheus metrics

A record produced by a source

A structured error for SourceReader::get_next_message implementors. Also implements From<anyhow::Error> for convenience.

A SourceToken manages interest in a source.


A wrapper around SourceMessage that allows SourceReader’s to communicate if a message if the final message a specific offset

The status of a source.


Types that implement this trait expose a length function

This trait defines the interface between Materialize and external sources, and must be implemented for every new kind of source.


Creates a raw source dataflow operator from a connection that has a corresponding SourceReader implentation. The type of ExternalSourceConnection determines the type of connection that should be created.

Returns true if the given source id/worker id is responsible for handling the given partition.