Module mz_storage::upsert
source · Modules§
- autospill πAn
UpsertStateBackend
that starts in memory and spills to RocksDB when the total size passes some threshold. - memory πAn
UpsertStateBackend
that stores values in memory. - rocksdb πAn
UpsertStateBackend
that stores values in RocksDB. - types πState-management for UPSERT.
Structs§
- Digest
Hasher π - Upsert
Config π
Enums§
- Drain
Style πThe style of drain we are performing on the stash.AtTime
-drains cannot assume that all values have been seen, and must leave tombstones behind for deleted values.
Traits§
- Upsert
Error πEmitter
Functions§
- drain_
staged_ πinput Helper method forupsert_inner
used to stagedata
updates from the input timely edge. - Emit the given error, and stall till the dataflow is restarted.
- This leaf operator drops
token
after the input reaches theresume_upper
. This is useful to take coordinated actions across all workers, after theupsert
operator has rehydrated. - stage_
input πHelper method forupsert_inner
used to stagedata
updates from the input timely edge. - upsert πResumes an upsert computation at
resume_upper
given as inputs a collection of upsert commands and the collection of the previous output of this operator. Returns a tuple of - upsert_
inner π - upsert_
thinning πRenders an operator that discards updates that are known to not affect the outcome of upsert in a streaming fashion. For each distinct (key, time) in the input it emits the value with the highest from_time. Its purpose is to thin out data as much as possible before exchanging them across workers.
Type Aliases§
- KeyHash πThe hash function used to map upsert keys. It is important that this hash is a cryptographic hash so that there is no risk of collisions. Collisions on SHA256 have a probability of 2^128 which is many orders of magnitude smaller than many other events that we donβt even think about (e.g bit flips). In short, we can safely assume that sha256(a) == sha256(b) iff a == b.