Struct Batch

pub struct Batch<K, V, T, D> {
    pub(crate) batch_delete_enabled: bool,
    pub(crate) metrics: Arc<Metrics>,
    pub(crate) shard_metrics: Arc<ShardMetrics>,
    pub(crate) version: Version,
    pub(crate) batch: HollowBatch<T>,
    pub(crate) blob: Arc<dyn Blob>,
    pub(crate) _phantom: PhantomData<fn() -> (K, V, T, D)>,
}

A handle to a batch of updates that has been written to blob storage but which has not yet been appended to a shard.

A Batch needs to be marked as consumed or it needs to be deleted via Self::delete. Otherwise, a dangling batch will leak and backing blobs will remain in blob storage.

Fields

batch_delete_enabled: bool

metrics: Arc<Metrics>

shard_metrics: Arc<ShardMetrics>

version: Version

The version of Materialize which wrote this batch.

batch: HollowBatch<T>

A handle to the data represented by this batch.

blob: Arc<dyn Blob>

Handle to the Blob that the blobs of this batch were uploaded to.

_phantom: PhantomData<fn() -> (K, V, T, D)>

Implementations

impl<K, V, T, D> Batch<K, V, T, D>

where K: Debug + Codec, V: Debug + Codec, T: Timestamp + Lattice + Codec64, D: Semigroup + Codec64,

pub(crate) fn new( batch_delete_enabled: bool, metrics: Arc<Metrics>, blob: Arc<dyn Blob>, shard_metrics: Arc<ShardMetrics>, version: Version, batch: HollowBatch<T>, ) -> Self

pub fn shard_id(&self) -> ShardId

The shard_id of this Batch.

pub fn upper(&self) -> &Antichain<T>

The upper of this Batch.

pub fn lower(&self) -> &Antichain<T>

The lower of this Batch.

pub(crate) fn mark_consumed(&mut self)

Marks the blobs that this batch handle points to as consumed, likely because they were appended to a shard.

Consumers of a blob need to make this explicit, so that we can log warnings in case a batch is not used.

pub async fn delete(self)

Deletes the blobs that make up this batch from the given blob store and marks them as deleted.

pub fn schemas(&self) -> impl Iterator<Item = SchemaId> + '_

Returns the schemas of parts in this batch.

pub fn into_hollow_batch(self) -> HollowBatch<T>

Turns this Batch into a HollowBatch.

NOTE: If this batch is not eventually appended to a shard or dropped, the data that it represents will have leaked.

pub fn into_transmittable_batch(self) -> ProtoBatch

Turns this Batch into a ProtoBatch, which can be used to transfer this batch across process boundaries, for example when exchanging data between timely workers.

NOTE: If this batch is not eventually appended to a shard or dropped, the data that it represents will have leaked. The caller is responsible for turning this back into a Batch using WriteHandle::batch_from_transmittable_batch.

pub(crate) async fn flush_to_blob( &mut self, cfg: &BatchBuilderConfig, batch_metrics: &BatchWriteMetrics, isolated_runtime: &Arc<IsolatedRuntime>, write_schemas: &Schemas<K, V>, )

impl<K, V, T, D> Batch<K, V, T, D>

where K: Debug + Codec, V: Debug + Codec, T: Timestamp + Lattice + Codec64 + TotalOrder, D: Semigroup + Codec64,

pub fn rewrite_ts( &mut self, frontier: &Antichain<T>, new_upper: Antichain<T>, ) -> Result<(), InvalidUsage<T>>

Efficiently rewrites the timestamps in this not-yet-committed batch.

This Batch represents potentially large amounts of data, which may have partly or entirely been spilled to s3. This call bulk edits the timestamps of all data in this batch in a metadata-only operation (i.e. without network calls).

Specifically, every timestamp in the batch is logically advanced_by the provided frontier.

This method may be called multiple times, with later calls overriding previous ones, but the rewrite frontier may not regress across calls.

When this batch was created, it was given an upper, which bounds the staged data it represents. To allow rewrite past this original upper, this call accepts a new upper which replaces the previous one. Like the rewrite frontier, the upper may not regress across calls.

Multiple batches with various rewrite frontiers may be used in a single crate::write::WriteHandle::compare_and_append_batch call. This is an expected usage.

This feature requires that the timestamp impls TotalOrder. This is because we need to be able to verify that the contained data, after the rewrite forward operation, still respects the new upper. It turns out that, given the metadata persist currently collects during batch collection, this is possible for totally ordered times, but it’s known to be not possible for partially ordered times. It is believed that we could fix this by collecting different metadata in batch creation (e.g. the join of or an antichain of the original contained timestamps), but the experience of database-issues#7825 has shaken our confidence in our own abilities to reason about partially ordered times and anyway all the initial uses have totally ordered times.

Trait Implementations

impl<K: Debug, V: Debug, T: Debug, D: Debug> Debug for Batch<K, V, T, D>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<K, V, T, D> Drop for Batch<K, V, T, D>

fn drop(&mut self)

Executes the destructor for this type.