Trait mz_storage::upsert::types::UpsertStateBackend

pub trait UpsertStateBackend<T, O>
where
    T: 'static,
    O: 'static,
{
    // Required methods
    fn supports_merge(&self) -> bool;
    fn multi_put<'life0, 'async_trait, P>(
        &'life0 mut self,
        puts: P,
    ) -> Pin<Box<dyn Future<Output = Result<PutStats, Error>> + 'async_trait>>
       where P: IntoIterator<Item = (UpsertKey, PutValue<StateValue<T, O>>)> + 'async_trait,
             Self: 'async_trait,
             'life0: 'async_trait;
    fn multi_get<'r, 'life0, 'async_trait, G, R>(
        &'life0 mut self,
        gets: G,
        results_out: R,
    ) -> Pin<Box<dyn Future<Output = Result<GetStats, Error>> + 'async_trait>>
       where G: IntoIterator<Item = UpsertKey> + 'async_trait,
             R: IntoIterator<Item = &'r mut UpsertValueAndSize<T, O>> + 'async_trait,
             Self: 'async_trait,
             'r: 'async_trait,
             'life0: 'async_trait;
    fn multi_merge<'life0, 'async_trait, P>(
        &'life0 mut self,
        merges: P,
    ) -> Pin<Box<dyn Future<Output = Result<MergeStats, Error>> + 'async_trait>>
       where P: IntoIterator<Item = (UpsertKey, MergeValue<StateValue<T, O>>)> + 'async_trait,
             Self: 'async_trait,
             'life0: 'async_trait;
}

A trait that defines the fundamental primitives required by a state-backing of UpsertState.

Implementors of this trait are blind maps that associate keys and values. They need not understand the semantics of StateValue, tombstones, or anything else related to a correct upsert implementation. The singular exception to this is that they must produce accurate PutStats and GetStats. The reasoning for this is two-fold:

• efficiency: this avoids additional buffer allocation.
• value sizes: only the backend implementation understands the size of values as recorded

This must is not a correctness requirement (we won’t panic when emitting statistics), but rather a requirement to ensure the upsert operator is introspectable.

Required Methods

fn supports_merge(&self) -> bool

Whether this backend supports the multi_merge operation.

fn multi_put<'life0, 'async_trait, P>( &'life0 mut self, puts: P, ) -> Pin<Box<dyn Future<Output = Result<PutStats, Error>> + 'async_trait>>

where P: IntoIterator<Item = (UpsertKey, PutValue<StateValue<T, O>>)> + 'async_trait, Self: 'async_trait, 'life0: 'async_trait,

Insert or delete for all puts keys, prioritizing the last value for repeated keys.

The PutValue is guaranteed to have an accurate and up-to-date record of the metadata for existing value for the given key (if one existed), as reported by a previous call to multi_get.

PutStats must be populated correctly, according to these semantics:

• values_diff must record the difference in number of new non-tombstone values being inserted into the backend.
• tombstones_diff must record the difference in number of tombstone values being inserted into the backend.
• size_diff must record the change in size for the values being inserted/deleted/updated in the backend, regardless of whether the values are tombstones or not.

fn multi_get<'r, 'life0, 'async_trait, G, R>( &'life0 mut self, gets: G, results_out: R, ) -> Pin<Box<dyn Future<Output = Result<GetStats, Error>> + 'async_trait>>

where G: IntoIterator<Item = UpsertKey> + 'async_trait, R: IntoIterator<Item = &'r mut UpsertValueAndSize<T, O>> + 'async_trait, Self: 'async_trait, 'r: 'async_trait, 'life0: 'async_trait,

Get the gets keys, which must be unique, placing the results in results_out.

Panics if gets and results_out are not the same length.

fn multi_merge<'life0, 'async_trait, P>( &'life0 mut self, merges: P, ) -> Pin<Box<dyn Future<Output = Result<MergeStats, Error>> + 'async_trait>>

where P: IntoIterator<Item = (UpsertKey, MergeValue<StateValue<T, O>>)> + 'async_trait, Self: 'async_trait, 'life0: 'async_trait,

For each key in merges writes a ‘merge operand’ to the backend. The backend stores these merge operands and periodically calls the consolidating_merge_function to merge them into any existing value for each key. The backend will merge the merge operands in the order they are provided, and the merge function will always be run for a given key when a get operation is performed on that key, or when the backend decides to run the merge based on its own internal logic. This allows avoiding the read-modify-write method of updating many values to improve performance.

The MergeValue should include a diff field that represents the update diff for the value. This is used to estimate the overall size diff of the working set after the merge operands are merged by the backend sum[merges: m](m.diff * m.size).

MergeStats must be populated correctly, according to these semantics:

• written_merge_operands must record the number of merge operands written to the backend.
• size_written must record the total size of values written to the backend. Note that the size of the post-merge values are not known, so this is the size of the values written to the backend as merge operands.
• size_diff must record the estimated diff of the total size of the working set after the merge operands are merged by the backend.

Object Safety

This trait is not object safe.

Implementors

impl<T, O> UpsertStateBackend<T, O> for InMemoryHashMap<T, O>

where O: Clone + 'static, T: Clone + 'static,

impl<T, O> UpsertStateBackend<T, O> for RocksDB<T, O>

where O: Send + Sync + Serialize + DeserializeOwned + 'static, T: Send + Sync + Serialize + DeserializeOwned + 'static,

impl<T, O, F, Fut> UpsertStateBackend<T, O> for AutoSpillBackend<T, O, F>

where O: Clone + Send + Sync + Serialize + DeserializeOwned + 'static, T: Clone + Send + Sync + Serialize + DeserializeOwned + 'static, F: FnOnce() -> Fut + 'static, Fut: Future<Output = RocksDB<T, O>>,