Module mz_storage::upsert

source Β·

Modules§

  • autospill πŸ”’
    An UpsertStateBackend that starts in memory and spills to RocksDB when the total size passes some threshold.
  • memory πŸ”’
    An UpsertStateBackend that stores values in memory.
  • rocksdb πŸ”’
    An UpsertStateBackend that stores values in RocksDB.
  • types πŸ”’
    State-management for UPSERT.

Structs§

Enums§

  • DrainStyle πŸ”’
    The style of drain we are performing on the stash. AtTime-drains cannot assume that all values have been seen, and must leave tombstones behind for deleted values.

Traits§

Functions§

  • Helper method for upsert_inner used to stage data updates from the input timely edge.
  • Emit the given error, and stall till the dataflow is restarted.
  • This leaf operator drops token after the input reaches the resume_upper. This is useful to take coordinated actions across all workers, after the upsert operator has rehydrated.
  • stage_input πŸ”’
    Helper method for upsert_inner used to stage data updates from the input timely edge.
  • upsert πŸ”’
    Resumes an upsert computation at resume_upper given as inputs a collection of upsert commands and the collection of the previous output of this operator. Returns a tuple of
  • upsert_inner πŸ”’
  • upsert_thinning πŸ”’
    Renders an operator that discards updates that are known to not affect the outcome of upsert in a streaming fashion. For each distinct (key, time) in the input it emits the value with the highest from_time. Its purpose is to thin out data as much as possible before exchanging them across workers.

Type Aliases§

  • KeyHash πŸ”’
    The hash function used to map upsert keys. It is important that this hash is a cryptographic hash so that there is no risk of collisions. Collisions on SHA256 have a probability of 2^128 which is many orders of magnitude smaller than many other events that we don’t even think about (e.g bit flips). In short, we can safely assume that sha256(a) == sha256(b) iff a == b.