Expand description
Dyncfgs used by the compute layer.
Constants§
- COMPUTE_
APPLY_ COLUMN_ DEMANDS - When enabled, applies the column demands from a MapFilterProject onto the RelationDesc used to read out of Persist. This allows Persist to prune unneeded columns as a performance optimization.
- COMPUTE_
FLAT_ MAP_ FUEL - The amount of output the flat-map operator produces before yielding. Set to a high value to avoid yielding, or to a low value to yield frequently.
- COMPUTE_
LOGICAL_ BACKPRESSURE_ INFLIGHT_ SLACK - The slack to round observed timestamps up to.
- COMPUTE_
LOGICAL_ BACKPRESSURE_ MAX_ RETAINED_ CAPABILITIES - Maximal number of capabilities retained by the logical backpressure operator.
- COMPUTE_
REPLICA_ EXPIRATION_ OFFSET - The maximum lifetime of a replica configured as an offset to the replica start time. Used in temporal filters to drop diffs generated at timestamps beyond the expiration time.
- COMPUTE_
SERVER_ MAINTENANCE_ INTERVAL - The interval at which the compute server performs maintenance tasks.
- CONSOLIDATING_
VEC_ GROWTH_ DAMPENER - The term
n
in the growth rate1 + 1/(n + 1)
forConsolidatingVec
. The smallest value0
corresponds to the greatest allowed growth, of doubling. - COPY_
TO_ S3_ ARROW_ BUILDER_ BUFFER_ RATIO - See
src/storage-operators/src/s3_oneshot_sink/parquet.rs
for more details. - COPY_
TO_ S3_ MULTIPART_ PART_ SIZE_ BYTES - The size of each part in the multi-part upload to use when uploading files to S3.
- COPY_
TO_ S3_ PARQUET_ ROW_ GROUP_ FILE_ RATIO - See
src/storage-operators/src/s3_oneshot_sink/parquet.rs
for more details. - DATAFLOW_
MAX_ INFLIGHT_ BYTES - Maximum number of in-flight bytes emitted by persist_sources feeding dataflows.
- DATAFLOW_
MAX_ INFLIGHT_ BYTES_ CC - The “physical backpressure” of
compute_dataflow_max_inflight_bytes_cc
has been replaced in cc replicas by persist lgalloc and we intend to remove it once everything has switched to cc. In the meantime, this is a CYA to turn it back on if absolutely necessary. - ENABLE_
ACTIVE_ DATAFLOW_ CANCELATION - Whether to use
drop_dataflow
to actively cancel dataflows. - ENABLE_
COLUMNAR_ LGALLOC - Enable lgalloc for columnar.
- ENABLE_
COLUMNATION_ LGALLOC - Enable lgalloc for columnation.
- ENABLE_
COMPUTE_ LOGICAL_ BACKPRESSURE - Whether to apply logical backpressure in compute dataflows.
- ENABLE_
COMPUTE_ RENDER_ FUELED_ AS_ SPECIFIC_ COLLECTION - Whether to render
as_specific_collection
using a fueled flat-map operator. - ENABLE_
COMPUTE_ REPLICA_ EXPIRATION - Main switch to enable or disable replica expiration.
- ENABLE_
CORRECTION_ V2 - Whether rendering should use the new MV sink correction buffer implementation.
- ENABLE_
LGALLOC - Enable lgalloc.
- ENABLE_
LGALLOC_ EAGER_ RECLAMATION - Enable lgalloc’s eager memory return/reclamation feature.
- ENABLE_
MV_ APPEND_ SMEARING - Whether the MV sink should distribute appends among workers.
- ENABLE_
MZ_ JOIN_ CORE - Whether rendering should use
mz_join_core
rather than DD’sJoinCore::join_core
. - ENABLE_
MZ_ JOIN_ CORE_ V2 - Whether rendering should use
mz_join_core_v2
rather than DD’sJoinCore::join_core
. - ENABLE_
PEEK_ RESPONSE_ STASH - Whether to enable the peek response stash, for sending back large peek
responses. The response stash will only be used for results that exceed
compute_peek_response_stash_threshold_bytes
. - ENABLE_
TEMPORAL_ BUCKETING - Whether to enable temporal bucketing in compute.
- HYDRATION_
CONCURRENCY - The number of dataflows that may hydrate concurrently.
- LGALLOC_
BACKGROUND_ INTERVAL - The interval at which the background thread wakes.
- LGALLOC_
FILE_ GROWTH_ DAMPENER - Enable lgalloc’s eager memory return/reclamation feature.
- LGALLOC_
LIMITER_ BURST_ FACTOR - Burst factor to disk limit.
- LGALLOC_
LIMITER_ INTERVAL - Interval to run the lgalloc limiter. A zero duration disables the limiter.
- LGALLOC_
LIMITER_ USAGE_ BIAS - Bias to the lgalloc limiter usage factor.
- LGALLOC_
LIMITER_ USAGE_ FACTOR - Factor of the memory limit that lgalloc will be permitted to use before terminating the process.
- LGALLOC_
LOCAL_ BUFFER_ BYTES - Enable lgalloc’s eager memory return/reclamation feature.
- LGALLOC_
SLOW_ CLEAR_ BYTES - The bytes to reclaim (slow path) per size class, for each background thread activation.
- LINEAR_
JOIN_ YIELDING - The yielding behavior with which linear joins should be rendered.
- MEMORY_
LIMITER_ BURST_ FACTOR - Burst factor to memory limit.
- MEMORY_
LIMITER_ INTERVAL - Interval to run the memory limiter. A zero duration disables the limiter.
- MEMORY_
LIMITER_ USAGE_ BIAS - Bias to the memory limiter usage factor.
- MEMORY_
LIMITER_ USAGE_ FACTOR - Factor of the memory limit that the process will be permitted to use before terminating the process.
- PEEK_
RESPONSE_ STASH_ BATCH_ MAX_ RUNS - The target number of maximum runs in the batches written to the stash.
- PEEK_
RESPONSE_ STASH_ READ_ BATCH_ SIZE_ BYTES - The target size for batches of rows we read out of the peek stash.
- PEEK_
RESPONSE_ STASH_ READ_ MEMORY_ BUDGET_ BYTES - The memory budget for consolidating stashed peek responses in
environmentd
. - PEEK_
RESPONSE_ STASH_ THRESHOLD_ BYTES - The threshold for peek response size above which we should use the peek response stash. Only used if the peek response stash is enabled and if the query is “streamable” (roughly: doesn’t have an ORDER BY).
- PEEK_
STASH_ BATCH_ SIZE - The size of each batch, as number of rows, pumped from the peek result iterator when stashing peek responses.
- PEEK_
STASH_ NUM_ BATCHES - The number of batches to pump from the peek result iterator when stashing peek responses.
- TEMPORAL_
BUCKETING_ SUMMARY - The summary to apply to the frontier in temporal bucketing in compute.
Functions§
- all_
dyncfgs - Adds the full set of all compute
Config
s.