Module dataflow::source [−][src]
Expand description
Types related to the creation of dataflow sources.
Modules
Generated protobuf code and companion impls.
“Base” metrics used by all dataflow sources.
Functionality for creating S3 sources
Types and methods for managing timestamp assignment and invention in sources
External users will interact primarily with instances of a TimestampBindingRc object
which lets various source instances reading on the same worker coordinate about the
underlying TimestampBindingBox and give readers that are lagging behind the ability
to delay compaction.
Besides that, the only other bit of complexity in this code is the TimestampProposer object
which manages the collaborative invention of timestamps by several source instances all reading
from the same worker. The key idea is that since all source readers are assigned to the same
worker, only one of them will be reading at a given time, and that reader can either consult
the timestamp bindings generated by its peers if it is not the furthest ahead, or if it is
the furthest ahead, it can propose a new assingment of (partition, offset) -> timestamp that
its peers will respect.
Structs
The output of the decoding operator
Contains all information necessary to ingest data from file sources
Kafka-specific information about the event
Contains all information necessary to ingest data from Kafka
Contains all information necessary to ingest data from Kinesis
Partition-specific metrics, recorded to both Prometheus and a system table
Configuration for persistent timestamp bindings.
Information required to sync data from Postgres
Information required to sync data from PubNub
Information required to load data from S3
Shared configuration information for all source types.
The data that we send from Upsert to the decode process
Source-agnostic wrapper for messages. Each source must implement a conversion to Message.
Source-specific Prometheus metrics
A record produced by a source
Util for restoring persisted timestamps and rendering persistence operators.
A SourceToken manages interest in a source.
An active transaction at a particular point in time. An instance of this struct is provided to a source when calling start_tx() on its timestamper. This has the effect of freezing the timestamper clock while the data for the transaction is sent.
Enums
Strategies for streaming content from a file.
The status of a source.
Statics
Traits
Types that implement this trait expose a length function
Simple sources must implement this trait. Sources will then get created as part of the
create_source_simple function.
This trait defines the interface between Materialize and external sources, and must be implemented for every new kind of source.
Functions
Creates a source dataflow operator. The type of ExternalSourceConnector determines the type of source that should be created
Creates a source dataflow operator from a connector implementing SimpleSource
Take message and assign it the appropriate timestamps and push it into the
dataflow layer, if possible.
Updates the given timestamp_bindings_updater with any changes from the given
timestamp_histories and emits updates to bindings_output if there are in fact any changes.
Blocking logic to read from a file, intended for its own thread.
Returns true if the given source id/worker id is responsible for handling the given partition.