Module dyncfgs

Source
Expand description

Dyncfgs used by the storage layer. Despite their name, these can be used “statically” during rendering, or dynamically within timely operators.

Constants§

CLUSTER_SHUTDOWN_GRACE_PERIOD
When dataflows observe an invariant violation it is either due to a bug or due to the cluster being shut down. This configuration defines the amount of time to wait before panicking the process, which will register the invariant violation.
DELAY_SOURCES_PAST_REHYDRATION
Whether rendering should use mz_join_core rather than DD’s JoinCore::join_core. Configuration for basic hydration backpressure.
ENFORCE_EXTERNAL_ADDRESSES
Whether or not to enforce that external connection addresses are global (not private or local) when resolving them.
KAFKA_BUFFERED_EVENT_RESIZE_THRESHOLD_ELEMENTS
KAFKA_CLIENT_ID_ENRICHMENT_RULES
Rules for enriching the client.id property of Kafka clients with additional data.
KAFKA_DEFAULT_AWS_PRIVATELINK_ENDPOINT_IDENTIFICATION_ALGORITHM
KAFKA_POLL_MAX_WAIT
The maximum time we will wait before re-polling rdkafka to see if new partitions/data are available.
MYSQL_OFFSET_KNOWN_INTERVAL
Interval to fetch offset_known, from @gtid_executed
MYSQL_REPLICATION_HEARTBEAT_INTERVAL
Replication heartbeat interval requested from the MySQL server.
ORE_OVERFLOWING_BEHAVIOR
Configure mz-ore overflowing type behavior.
PG_FETCH_SLOT_RESUME_LSN_INTERVAL
Interval to poll confirmed_flush_lsn to get a resumption lsn.
PG_OFFSET_KNOWN_INTERVAL
Interval to fetch offset_known, from pg_current_wal_lsn
PG_SCHEMA_VALIDATION_INTERVAL
Interval to re-validate the schemas of ingested tables.
REPLICA_METRICS_HISTORY_RETENTION_INTERVAL
The interval of time to keep when truncating the replica metrics history.
SINK_ENSURE_TOPIC_CONFIG
Configure how to behave when trying to create an existing topic with specified configs.
SINK_PROGRESS_SEARCH
If set, iteratively search the progress topic for a progress record with increasing lookback.
STATISTICS_RETENTION_DURATION
The time after which we delete per-replica statistics (for sources and sinks) after there have been no updates.
STORAGE_DOWNGRADE_SINCE_DURING_FINALIZATION
When enabled, force-downgrade the controller’s since handle on the shard during shard finalization.
STORAGE_RECLOCK_TO_LATEST
Whether to mint reclock bindings based on the latest probed frontier or the currently ingested frontier.
STORAGE_ROCKSDB_CLEANUP_TRIES
How many times to try to cleanup old RocksDB DB’s on disk before giving up.
STORAGE_ROCKSDB_USE_MERGE_OPERATOR
Whether to enable the merge operator in upsert for the RocksDB backend.
STORAGE_SERVER_MAINTENANCE_INTERVAL
The interval at which the storage server performs maintenance tasks.
STORAGE_SINK_SNAPSHOT_FRONTIER
If true, skip fetching the snapshot in the sink once the frontier has advanced.
STORAGE_SUSPEND_AND_RESTART_DELAY
Delay interval when reconnecting to a source / sink after halt.
STORAGE_UPSERT_MAX_SNAPSHOT_BATCH_BUFFERING
If storage_upsert_prevent_snapshot_buffering is true, this prevents the upsert operator from buffering too many events from the upstream snapshot. In the absence of hydration flow control, this could prevent certain workloads from causing egregiously large writes to RocksDB.
STORAGE_UPSERT_PREVENT_SNAPSHOT_BUFFERING
Whether or not to prevent buffering the entire upstream snapshot in memory when processing it in memory. This is generally understood to reduce memory consumption.
STORAGE_USE_CONTINUAL_FEEDBACK_UPSERT
Whether to use the new continual feedback upsert operator.
SUSPENDABLE_SOURCES
Whether storage dataflows should suspend execution while downstream operators are still processing data.
WALLCLOCK_GLOBAL_LAG_HISTOGRAM_RETENTION_INTERVAL
The interval of time to keep when truncating the wallclock lag histogram.
WALLCLOCK_LAG_HISTORY_RETENTION_INTERVAL
The interval of time to keep when truncating the wallclock lag history.

Statics§

KAFKA_METADATA_FETCH_INTERVAL
Interval to fetch topic partition metadata.

Functions§

all_dyncfgs
Adds the full set of all storage Configs.