Trait dataflow::source::SourceReader [−][src]
pub(crate) trait SourceReader {
type Key: Data + MaybeLength;
type Value: Data + MaybeLength;
fn new(
source_name: String,
source_id: SourceInstanceId,
worker_id: usize,
worker_count: usize,
consumer_activator: SyncActivator,
connector: ExternalSourceConnector,
restored_offsets: Vec<(PartitionId, Option<MzOffset>)>,
encoding: SourceDataEncoding,
logger: Option<Logger>,
metrics: SourceBaseMetrics
) -> Result<(Self, Option<PartitionId>), Error>
where
Self: Sized;
fn get_next_message(
&mut self
) -> Result<NextMessage<Self::Key, Self::Value>, Error>;
}
Expand description
This trait defines the interface between Materialize and external sources, and must be implemented for every new kind of source.
TODO: this trait is still a little too Kafka-centric, specifically the concept of a “partition” is baked into this trait and introduces some cognitive overhead as we are forced to treat things like file sources as “single-partition”
Associated Types
type Key: Data + MaybeLength
type Value: Data + MaybeLength
Required methods
fn new(
source_name: String,
source_id: SourceInstanceId,
worker_id: usize,
worker_count: usize,
consumer_activator: SyncActivator,
connector: ExternalSourceConnector,
restored_offsets: Vec<(PartitionId, Option<MzOffset>)>,
encoding: SourceDataEncoding,
logger: Option<Logger>,
metrics: SourceBaseMetrics
) -> Result<(Self, Option<PartitionId>), Error> where
Self: Sized,
fn new(
source_name: String,
source_id: SourceInstanceId,
worker_id: usize,
worker_count: usize,
consumer_activator: SyncActivator,
connector: ExternalSourceConnector,
restored_offsets: Vec<(PartitionId, Option<MzOffset>)>,
encoding: SourceDataEncoding,
logger: Option<Logger>,
metrics: SourceBaseMetrics
) -> Result<(Self, Option<PartitionId>), Error> where
Self: Sized,
Create a new source reader.
This function returns the source reader and optionally, any “partition” it’s already reading. In practice, the partition is only non-None for static sources that either don’t truly have partitions or have a fixed number of partitions.
fn get_next_message(
&mut self
) -> Result<NextMessage<Self::Key, Self::Value>, Error>
fn get_next_message(
&mut self
) -> Result<NextMessage<Self::Key, Self::Value>, Error>
Returns the next message available from the source.
Note that implementers are required to present messages in strictly ascending
offset order within each partition.