Module dyncfgs

Dyncfgs used by the compute layer.

Constants

COMPUTE_APPLY_COLUMN_DEMANDS

When enabled, applies the column demands from a MapFilterProject onto the RelationDesc used to read out of Persist. This allows Persist to prune unneeded columns as a performance optimization.

COMPUTE_FLAT_MAP_FUEL

The amount of output the flat-map operator produces before yielding. Set to a high value to avoid yielding, or to a low value to yield frequently.

COMPUTE_LOGICAL_BACKPRESSURE_INFLIGHT_SLACK

The slack to round observed timestamps up to.

COMPUTE_LOGICAL_BACKPRESSURE_MAX_RETAINED_CAPABILITIES

Maximal number of capabilities retained by the logical backpressure operator.

COMPUTE_REPLICA_EXPIRATION_OFFSET

The maximum lifetime of a replica configured as an offset to the replica start time. Used in temporal filters to drop diffs generated at timestamps beyond the expiration time.

COMPUTE_SERVER_MAINTENANCE_INTERVAL

The interval at which the compute server performs maintenance tasks.

CONSOLIDATING_VEC_GROWTH_DAMPENER

The term n in the growth rate 1 + 1/(n + 1) for ConsolidatingVec. The smallest value 0 corresponds to the greatest allowed growth, of doubling.

COPY_TO_S3_ARROW_BUILDER_BUFFER_RATIO

See src/storage-operators/src/s3_oneshot_sink/parquet.rs for more details.

COPY_TO_S3_MULTIPART_PART_SIZE_BYTES

The size of each part in the multi-part upload to use when uploading files to S3.

COPY_TO_S3_PARQUET_ROW_GROUP_FILE_RATIO

See src/storage-operators/src/s3_oneshot_sink/parquet.rs for more details.

DATAFLOW_MAX_INFLIGHT_BYTES

Maximum number of in-flight bytes emitted by persist_sources feeding dataflows.

DATAFLOW_MAX_INFLIGHT_BYTES_CC

The “physical backpressure” of compute_dataflow_max_inflight_bytes_cc has been replaced in cc replicas by persist lgalloc and we intend to remove it once everything has switched to cc. In the meantime, this is a CYA to turn it back on if absolutely necessary.

ENABLE_ACTIVE_DATAFLOW_CANCELATION

Whether to use drop_dataflow to actively cancel dataflows.

ENABLE_COLUMNAR_LGALLOC

Enable lgalloc for columnar.

ENABLE_COLUMNATION_LGALLOC

Enable lgalloc for columnation.

ENABLE_COMPUTE_LOGICAL_BACKPRESSURE

Whether to apply logical backpressure in compute dataflows.

ENABLE_COMPUTE_RENDER_FUELED_AS_SPECIFIC_COLLECTION

Whether to render as_specific_collection using a fueled flat-map operator.

ENABLE_COMPUTE_REPLICA_EXPIRATION

Main switch to enable or disable replica expiration.

ENABLE_CORRECTION_V2

Whether rendering should use the new MV sink correction buffer implementation.

ENABLE_LGALLOC

Enable lgalloc.

ENABLE_LGALLOC_EAGER_RECLAMATION

Enable lgalloc’s eager memory return/reclamation feature.

ENABLE_MZ_JOIN_CORE

Whether rendering should use mz_join_core rather than DD’s JoinCore::join_core.

ENABLE_MZ_JOIN_CORE_V2

Whether rendering should use mz_join_core_v2 rather than DD’s JoinCore::join_core.

ENABLE_PEEK_RESPONSE_STASH

Whether to enable the peek response stash, for sending back large peek responses. The response stash will only be used for results that exceed compute_peek_response_stash_threshold_bytes.

ENABLE_TEMPORAL_BUCKETING

Whether to enable temporal bucketing in compute.

HYDRATION_CONCURRENCY

The number of dataflows that may hydrate concurrently.

LGALLOC_BACKGROUND_INTERVAL

The interval at which the background thread wakes.

LGALLOC_FILE_GROWTH_DAMPENER

Enable lgalloc’s eager memory return/reclamation feature.

LGALLOC_LOCAL_BUFFER_BYTES

Enable lgalloc’s eager memory return/reclamation feature.

LGALLOC_SLOW_CLEAR_BYTES

The bytes to reclaim (slow path) per size class, for each background thread activation.

LINEAR_JOIN_YIELDING

The yielding behavior with which linear joins should be rendered.

MEMORY_LIMITER_BURST_FACTOR

Burst factor to memory limit.

MEMORY_LIMITER_INTERVAL

Interval to run the memory limiter. A zero duration disables the limiter.

MEMORY_LIMITER_USAGE_BIAS

Bias to the memory limiter usage factor.

PEEK_RESPONSE_STASH_BATCH_MAX_RUNS

The target number of maximum runs in the batches written to the stash.

PEEK_RESPONSE_STASH_READ_BATCH_SIZE_BYTES

The target size for batches of rows we read out of the peek stash.

PEEK_RESPONSE_STASH_READ_MEMORY_BUDGET_BYTES

The memory budget for consolidating stashed peek responses in environmentd.

PEEK_RESPONSE_STASH_THRESHOLD_BYTES

The threshold for peek response size above which we should use the peek response stash. Only used if the peek response stash is enabled and if the query is “streamable” (roughly: doesn’t have an ORDER BY).

PEEK_STASH_BATCH_SIZE

The size of each batch, as number of rows, pumped from the peek result iterator when stashing peek responses.

PEEK_STASH_NUM_BATCHES

The number of batches to pump from the peek result iterator when stashing peek responses.

TEMPORAL_BUCKETING_SUMMARY

The summary to apply to the frontier in temporal bucketing in compute.

Functions

all_dyncfgs

Adds the full set of all compute Configs.