Module mz_adapter::coord

Translation of SQL commands into timestamped Controller commands.

The various SQL commands instruct the system to take actions that are not yet explicitly timestamped. On the other hand, the underlying data continually change as time moves forward. On the third hand, we greatly benefit from the information that some times are no longer of interest, so that we may compact the representation of the continually changing collections.

The Coordinator curates these interactions by observing the progress collections make through time, choosing timestamps for its own commands, and eventually communicating that certain times have irretrievably “passed”.

Frontiers another way

If the above description of frontiers left you with questions, this repackaged explanation might help.

• since is the least recent time (i.e. oldest time) that you can read from sources and be guaranteed that the returned data is accurate as of that time.

  Reads at times less than since may return values that were not actually seen at the specified time, but arrived later (i.e. the results are compacted).

  For correctness’ sake, the coordinator never chooses to read at a time less than an arrangement’s since.

• upper is the first time after the most recent time that you can read from sources and receive an immediate response. Alternately, it is the least time at which the data may still change (that is the reason we may not be able to respond immediately).

  Reads at times >= upper may not immediately return because the answer isn’t known yet. However, once the upper is > the specified read time, the read can return.

  For the sake of returned values’ freshness, the coordinator prefers performing reads at an arrangement’s upper. However, because we more strongly prefer correctness, the coordinator will choose timestamps greater than an object’s upper if it is also being accessed alongside objects whose since times are >= its upper.

This illustration attempts to show, with time moving left to right, the relationship between since and upper.

• #: possibly inaccurate results
• -: immediate, correct response
• ?: not yet known
• s: since
• u: upper
• |: eligible for coordinator to select

####s----u?????
    |||||||||||

Modules

• appends 🔒
  Logic and types for all appends executed by the Coordinator.
• catalog_serving 🔒
  Special cases related to the “catalog serving” of Materialize
• caught_up 🔒
  Support for checking whether clusters/collections are caught up during a 0dt deployment.
• cluster_scheduling
• command_handler 🔒
  Logic for processing client Commands. Each Command is initiated by a client via some external Materialize API (ex: HTTP and psql).
• consistency
  Internal consistency checks that validate invariants of Coordinator.
• ddl 🔒
  This module encapsulates all of the Coordinator’s logic for creating, dropping, and altering objects.
• id_bundle 🔒
• in_memory_oracle 🔒
  A timestamp oracle that relies on the crate::catalog::Catalog for persistence/durability and reserves ranges of timestamps.
• indexes 🔒
• introspection 🔒
  Support for unified compute introspection.
• message_handler 🔒
  Logic for processing Coordinator messages. The Coordinator receives messages from various sources (ex: controller, clients, background tasks, etc).
• peek 🔒
  Logic and types for creating, executing, and tracking peeks.
• privatelink_status 🔒
• read_policy
  Types and methods related to initializing, updating, and removing read policies on collections.
• sequencer 🔒
  Logic for executing a planned SQL query.
• sql 🔒
  Various utility methods used by the Coordinator. Ideally these are all put in more meaningfully named modules.
• statement_logging 🔒
• timeline 🔒
  A mechanism to ensure that a sequence of writes and reads proceed correctly through timestamps.
• timestamp_selection 🔒
  Logic for selecting timestamps for various operations on collections.
• validity 🔒

Structs

• AlterCluster
• AlterClusterFinalize
• AlterClusterWaitForHydrated
• AlterSecret
• AlterSinkReadyContext 🔒
• BackgroundWorkResult
• ClusterReplicaStatuses 🔒
• Config
  Configures a coordinator.
• ConnMeta
  Metadata about an active connection.
• Coordinator
  Glues the external world to the Timely workers.
• CopyToContext
• CreateIndexExplain
• CreateIndexFinish
• CreateIndexOptimize
• CreateMaterializedViewExplain
• CreateMaterializedViewFinish
• CreateMaterializedViewOptimize
• CreateSecretEnsure
• CreateSecretFinish
• CreateViewExplain
• CreateViewFinish
• CreateViewOptimize
• DeferredPlanStatement 🔒
• ExecuteContext
  Bundle of state related to statement execution.
• ExecuteContextExtra
  State that the coordinator must process as part of retiring command execution. ExecuteContextExtra::Default is guaranteed to produce a value that will cause the coordinator to do nothing, and is intended for use by code that invokes the execution processing flow (i.e., sequence_plan) without actually being a statement execution.
• ExecuteContextInner
• ExplainPlanContext
• ExplainTimestampFinish
• ExplainTimestampOptimize
• ExplainTimestampRealTimeRecency
• IntrospectionSubscribeFinish
• IntrospectionSubscribeOptimizeMir
• IntrospectionSubscribeTimestampOptimizeLir
• LastMessage 🔒
  Contains information about the last message the Coordinator processed.
• LockedVecDeque 🔒
  A struct for tracking the ownership of a lock and a VecDeque to store to-be-done work after the lock is freed.
• PeekStageCopyTo
• PeekStageExplainPlan
• PeekStageExplainPushdown
• PeekStageFinish
• PeekStageLinearizeTimestamp
• PeekStageOptimize
• PeekStageRealTimeRecency
• PeekStageTimestampReadHold
• PendingReadTxn
  A pending read transaction waiting to be linearized along with metadata about it’s state
• PendingTxn
  A pending transaction waiting to be committed.
• ReplicaMetadata
  Soft-state metadata about a compute replica
• RotateKeysSecretEnsure
• RotateKeysSecretFinish
• SubscribeFinish
• SubscribeOptimizeMir
• SubscribeTimestampOptimizeLir
• ValidationReady

Enums

• ClusterStage
• CreateIndexStage
• CreateMaterializedViewStage
• CreateViewStage
• ExplainContext
• ExplainTimestampStage
• IntrospectionSubscribeStage
• Message
• NetworkPolicyError
• PeekStage
• PendingRead 🔒
  A pending read transaction waiting to be linearized.
• PendingTxnResponse
  The response we’ll send for a PendingTxn.
• PlanStatement 🔒
• SecretStage
• StageResult 🔒
  Result types for each stage of a sequence.
• SubscribeStage
• TargetCluster
  An enum describing which cluster to run a statement on.
• WatchSetResponse 🔒

Traits

• Staged 🔒
  Common functionality for Coordinator::sequence_staged.
• StagedContext

Functions

• get_initial_oracle_timestamps 🔒
• load_remote_system_parameters
• serve
  Serves the coordinator based on the provided configuration.
• validate_ip_with_policy_rules 🔒

Type Aliases

• AlterConnectionValidationReady
• CreateConnectionValidationReady
• PurifiedStatementReady