Struct StashUpload 

pub(super) struct StashUpload {
    relation_desc: RelationDesc,
    shard_id: ShardId,
    batch_builder: Option<BatchBuilder<SourceData, (), Timestamp, i64>>,
    upper: Antichain<Timestamp>,
    num_rows: u64,
    wrote_parts: bool,
    runtime: Handle,
}

A peek’s answer on its way to the peek stash, written to persist a batch of rows at a time.

The upload owns the IO the stash needs, so a walk that feeds it performs none: a driver that can await pushes the rows the walk produced and finishes the upload, and the walk itself neither opens a client nor writes a byte. That split keeps a walk drivable from a timely worker and from an async task alike.

The rows an upload is given are the rows it writes, in the order it is given them. An upload that does not reach a reader deletes what it can, whether it is dropped by a driver that will answer with something else or stopped part-way through finishing. StashUpload::abandon bounds what that reaches and what it costs.

Fields

relation_desc: RelationDesc

The description the stashed response reports, and the schema the batch is written under.

shard_id: ShardId

The shard the batch belongs to, derived from the peek’s uuid so that a reader holding the response can find it.

batch_builder: Option<BatchBuilder<SourceData, (), Timestamp, i64>>

The parts persist has taken so far. Taken by whichever of StashUpload::finish and StashUpload::abandon gets there first, and a None says the parts are accounted for and nothing is left to delete.

upper: Antichain<Timestamp>

The upper the batch is finished at, one step beyond the timestamp every row is written at.

num_rows: u64

Rows written so far, counting a row with a diff of n as n rows, which is how the finishing counts them.

wrote_parts: bool

Whether persist has taken a part off this builder and into blob storage. Until it has, everything the upload holds is in memory, so an abandoned upload drops its builder instead of paying a write and a delete to reclaim nothing.

runtime: Handle

The runtime an abandoned upload’s deletion is spawned on, held rather than taken from the ambient context because StashUpload::abandon runs where there may be none: a Drop carries no runtime context of its own, and Handle::current panics without one.

Implementations

impl StashUpload

pub(super) async fn open( persist_clients: &PersistClientCache, persist_location: PersistLocation, batch_max_runs: usize, peek_uuid: Uuid, relation_desc: RelationDesc, ) -> Result<Self, StashError>

Opens an upload for the peek peek_uuid identifies.

Fails when the stash location does not open, in which case nothing has been written.

pub(super) async fn push( &mut self, rows: Vec<(Row, NonZeroI64)>, ) -> Result<(), StashError>

Writes rows to the stash.

Every row given is written. Stopping where the peek’s finishing has all it can use is the scan’s to decide, since it is the scan that holds the finishing and produces the rows.

Fails where persist rejects the write, which leaves the upload unusable and the rows it holds unanswerable.

pub(super) async fn finish(self) -> Result<PeekResponse, StashError>

Finishes the batch and builds the response that names it.

Every row of the answer is in the batch. The response’s inline_rows are left for the controller, which merges in the rows of workers whose share never reached the stash.

Fails where persist rejects the batch, which takes the parts with it: persist keeps the builder and hands back no handle to what it holds. Only a batch whose bounds do not admit its own updates is refused, which this upload’s fixed lower, upper and timestamp cannot produce.

async fn finish_batch(&mut self) -> Result<DeliveredBatch, StashError>

Finishes the batch as work of its own, and leaves the upload holding no parts.

Flushing the buffered part and the uploads still in flight is the longest await an upload makes, so a cancellation most likely lands there, with the builder already out of the upload. A task the cancellation cannot reach holds it instead, and whoever ends up holding a batch nobody will read deletes it.

fn abandon(&mut self)

Schedules the deletion of whatever parts the upload still holds, and leaves it holding none.

Persist hands back a deletable handle only by finishing the batch, so the buffered rows are written out first. That write reaches persist_blob_target_size, 128 MiB by default, and nothing bounds how many abandoned uploads carry one at once, because the walk releases its permit before this finishes.

The handle does not reach parts a run merge already dropped from shard state. A reader deleting a response it has finished with reaches the same set, so that bounds the builder rather than abandonment.

TODO: a builder teardown that surrendered the written parts without flushing the buffered one would make this cost a delete and nothing else.

Trait Implementations

impl Drop for StashUpload

fn drop(&mut self)

Deletes the parts of an upload that ends without StashUpload::finish, which is the only cleanup a walk aborted mid-upload gets.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.