Constant mz_storage_types::dyncfgs::STORAGE_UPSERT_PREVENT_SNAPSHOT_BUFFERING

pub const STORAGE_UPSERT_PREVENT_SNAPSHOT_BUFFERING: Config<bool>;

Whether or not to prevent buffering the entire upstream snapshot in memory when processing it in memory. This is generally understood to reduce memory consumption.

When false, in general the memory utilization while processing the snapshot is:

of snapshot updates + (# of unique keys in snapshot * N), where N is some small

integer number of buffers

When true, in general the memory utilization while processing the snapshot is:

of snapshot updates + (RocksDB buffers + # of keys in batch produced by upstream) * # of

workers.

Without hydration flow control, which is not yet implemented, there are workloads that may cause the latter to use more memory, which is why we offer this configuration.