Struct TransactionalSnapshotProducer

struct TransactionalSnapshotProducer {
    pi: PartitionKeyIterator,
    batch_size: u64,
    produced_batches: u64,
    expected_batches: u64,
    round: u64,
    snapshot_rounds: u64,
    rng: Option<StdRng>,
    seed: u64,
    include_offset: bool,
}

A struct that produces batches of data for the snapshotting phase.

Fields

pi: PartitionKeyIterator

The key iterator for the partition.

batch_size: u64

The batch size.

produced_batches: u64

The number of batches produced in the current round.

expected_batches: u64

The expected number of batches per round.

round: u64

The current round of the offset

snapshot_rounds: u64

The total number of rounds.

rng: Option<StdRng>

The rng for the current round.

seed: u64

The source-level seed for the rng.

include_offset: bool

Whether to include the offset or not.

Implementations

impl TransactionalSnapshotProducer

fn new(partition: u64, key_value: KeyValueLoadGenerator) -> Self

fn finished(&self) -> bool

If this partition is done snapshotting.

fn produce_batch<'a>( &'a mut self, value_buffer: &'a mut Vec<u8>, output_indexes: &'a [usize], ) -> impl Iterator<Item = ((usize, Result<SourceMessage, DataflowError>), MzOffset, Diff)> + 'a

Produce a batch of message into buffer, of size batch_size. Advances the current batch and round counter.

The output iterator must be consumed fully for this method to be used correctly.