pub const STORAGE_UPSERT_PREVENT_SNAPSHOT_BUFFERING: Config<bool>;
Expand description
Whether or not to prevent buffering the entire upstream snapshot in memory when processing it in memory. This is generally understood to reduce memory consumption.
When false, in general the memory utilization while processing the snapshot is:
§of snapshot updates + (# of unique keys in snapshot * N), where N is some small
integer number of buffers
When true, in general the memory utilization while processing the snapshot is:
§of snapshot updates + (RocksDB buffers + # of keys in batch produced by upstream) * # of
workers.
Without hydration flow control, which is not yet implemented, there are workloads that may cause the latter to use more memory, which is why we offer this configuration.