Module dyncfgs

Source
Expand description

Dyncfgs used by the compute layer.

Constants§

COMPUTE_APPLY_COLUMN_DEMANDS
When enabled, applies the column demands from a MapFilterProject onto the RelationDesc used to read out of Persist. This allows Persist to prune unneeded columns as a performance optimization.
COMPUTE_FLAT_MAP_FUEL
The amount of output the flat-map operator produces before yielding. Set to a high value to avoid yielding, or to a low value to yield frequently.
COMPUTE_LOGICAL_BACKPRESSURE_INFLIGHT_SLACK
The slack to round observed timestamps up to.
COMPUTE_LOGICAL_BACKPRESSURE_MAX_RETAINED_CAPABILITIES
Maximal number of capabilities retained by the logical backpressure operator.
COMPUTE_REPLICA_EXPIRATION_OFFSET
The maximum lifetime of a replica configured as an offset to the replica start time. Used in temporal filters to drop diffs generated at timestamps beyond the expiration time.
COMPUTE_SERVER_MAINTENANCE_INTERVAL
The interval at which the compute server performs maintenance tasks.
CONSOLIDATING_VEC_GROWTH_DAMPENER
The term n in the growth rate 1 + 1/(n + 1) for ConsolidatingVec. The smallest value 0 corresponds to the greatest allowed growth, of doubling.
COPY_TO_S3_ARROW_BUILDER_BUFFER_RATIO
See src/storage-operators/src/s3_oneshot_sink/parquet.rs for more details.
COPY_TO_S3_MULTIPART_PART_SIZE_BYTES
The size of each part in the multi-part upload to use when uploading files to S3.
COPY_TO_S3_PARQUET_ROW_GROUP_FILE_RATIO
See src/storage-operators/src/s3_oneshot_sink/parquet.rs for more details.
DATAFLOW_MAX_INFLIGHT_BYTES
Maximum number of in-flight bytes emitted by persist_sources feeding dataflows.
DATAFLOW_MAX_INFLIGHT_BYTES_CC
The “physical backpressure” of compute_dataflow_max_inflight_bytes_cc has been replaced in cc replicas by persist lgalloc and we intend to remove it once everything has switched to cc. In the meantime, this is a CYA to turn it back on if absolutely necessary.
ENABLE_ACTIVE_DATAFLOW_CANCELATION
Whether to use drop_dataflow to actively cancel dataflows.
ENABLE_COLUMNAR_LGALLOC
Enable lgalloc for columnar.
ENABLE_COLUMNATION_LGALLOC
Enable lgalloc for columnation.
ENABLE_COMPUTE_LOGICAL_BACKPRESSURE
Whether to apply logical backpressure in compute dataflows.
ENABLE_COMPUTE_RENDER_FUELED_AS_SPECIFIC_COLLECTION
Whether to render as_specific_collection using a fueled flat-map operator.
ENABLE_COMPUTE_REPLICA_EXPIRATION
Main switch to enable or disable replica expiration.
ENABLE_CORRECTION_V2
Whether rendering should use the new MV sink correction buffer implementation.
ENABLE_LGALLOC
Enable lgalloc.
ENABLE_LGALLOC_EAGER_RECLAMATION
Enable lgalloc’s eager memory return/reclamation feature.
ENABLE_MV_APPEND_SMEARING
Whether the MV sink should distribute appends among workers.
ENABLE_MZ_JOIN_CORE
Whether rendering should use mz_join_core rather than DD’s JoinCore::join_core.
ENABLE_MZ_JOIN_CORE_V2
Whether rendering should use mz_join_core_v2 rather than DD’s JoinCore::join_core.
ENABLE_PEEK_RESPONSE_STASH
Whether to enable the peek response stash, for sending back large peek responses. The response stash will only be used for results that exceed compute_peek_response_stash_threshold_bytes.
ENABLE_TEMPORAL_BUCKETING
Whether to enable temporal bucketing in compute.
HYDRATION_CONCURRENCY
The number of dataflows that may hydrate concurrently.
LGALLOC_BACKGROUND_INTERVAL
The interval at which the background thread wakes.
LGALLOC_FILE_GROWTH_DAMPENER
Enable lgalloc’s eager memory return/reclamation feature.
LGALLOC_LIMITER_BURST_FACTOR
Burst factor to disk limit.
LGALLOC_LIMITER_INTERVAL
Interval to run the lgalloc limiter. A zero duration disables the limiter.
LGALLOC_LIMITER_USAGE_BIAS
Bias to the lgalloc limiter usage factor.
LGALLOC_LIMITER_USAGE_FACTOR
Factor of the memory limit that lgalloc will be permitted to use before terminating the process.
LGALLOC_LOCAL_BUFFER_BYTES
Enable lgalloc’s eager memory return/reclamation feature.
LGALLOC_SLOW_CLEAR_BYTES
The bytes to reclaim (slow path) per size class, for each background thread activation.
LINEAR_JOIN_YIELDING
The yielding behavior with which linear joins should be rendered.
MEMORY_LIMITER_BURST_FACTOR
Burst factor to memory limit.
MEMORY_LIMITER_INTERVAL
Interval to run the memory limiter. A zero duration disables the limiter.
MEMORY_LIMITER_USAGE_BIAS
Bias to the memory limiter usage factor.
MEMORY_LIMITER_USAGE_FACTOR
Factor of the memory limit that the process will be permitted to use before terminating the process.
PEEK_RESPONSE_STASH_BATCH_MAX_RUNS
The target number of maximum runs in the batches written to the stash.
PEEK_RESPONSE_STASH_READ_BATCH_SIZE_BYTES
The target size for batches of rows we read out of the peek stash.
PEEK_RESPONSE_STASH_READ_MEMORY_BUDGET_BYTES
The memory budget for consolidating stashed peek responses in environmentd.
PEEK_RESPONSE_STASH_THRESHOLD_BYTES
The threshold for peek response size above which we should use the peek response stash. Only used if the peek response stash is enabled and if the query is “streamable” (roughly: doesn’t have an ORDER BY).
PEEK_STASH_BATCH_SIZE
The size of each batch, as number of rows, pumped from the peek result iterator when stashing peek responses.
PEEK_STASH_NUM_BATCHES
The number of batches to pump from the peek result iterator when stashing peek responses.
TEMPORAL_BUCKETING_SUMMARY
The summary to apply to the frontier in temporal bucketing in compute.

Functions§

all_dyncfgs
Adds the full set of all compute Configs.