Module upsert

Modules§

autospill 🔒: An UpsertStateBackend that starts in memory and spills to RocksDB when the total size passes some threshold.
memory: An UpsertStateBackend that stores values in memory.
rocksdb 🔒: An UpsertStateBackend that stores values in RocksDB.
types 🔒: State-management for UPSERT.

DrainStyle 🔒: The style of drain we are performing on the stash. AtTime-drains cannot assume that all values have been seen, and must leave tombstones behind for deleted values.

drain_staged_input 🔒: Helper method for upsert_inner used to stage data updates from the input timely edge.
process_upsert_state_error 🔒: Emit the given error, and stall till the dataflow is restarted.
rehydration_finished: This leaf operator drops token after the input reaches the resume_upper. This is useful to take coordinated actions across all workers, after the upsert operator has rehydrated.
stage_input 🔒: Helper method for upsert_classic used to stage data updates from the input/source timely edge.
upsert 🔒: Resumes an upsert computation at resume_upper given as inputs a collection of upsert commands and the collection of the previous output of this operator. Returns a tuple of
upsert_classic 🔒
upsert_operator 🔒
upsert_thinning 🔒: Renders an operator that discards updates that are known to not affect the outcome of upsert in a streaming fashion. For each distinct (key, time) in the input it emits the value with the highest from_time. Its purpose is to thin out data as much as possible before exchanging them across workers.

KeyHash 🔒: The hash function used to map upsert keys. It is important that this hash is a cryptographic hash so that there is no risk of collisions. Collisions on SHA256 have a probability of 2^128 which is many orders of magnitude smaller than many other events that we don’t even think about (e.g bit flips). In short, we can safely assume that sha256(a) == sha256(b) iff a == b.
UpsertValue