Expand description
Translation of SQL commands into timestamped Controller
commands.
The various SQL commands instruct the system to take actions that are not yet explicitly timestamped. On the other hand, the underlying data continually change as time moves forward. On the third hand, we greatly benefit from the information that some times are no longer of interest, so that we may compact the representation of the continually changing collections.
The Coordinator
curates these interactions by observing the progress
collections make through time, choosing timestamps for its own commands,
and eventually communicating that certain times have irretrievably “passed”.
§Frontiers another way
If the above description of frontiers left you with questions, this repackaged explanation might help.
-
since
is the least recent time (i.e. oldest time) that you can read from sources and be guaranteed that the returned data is accurate as of that time.Reads at times less than
since
may return values that were not actually seen at the specified time, but arrived later (i.e. the results are compacted).For correctness’ sake, the coordinator never chooses to read at a time less than an arrangement’s
since
. -
upper
is the first time after the most recent time that you can read from sources and receive an immediate response. Alternately, it is the least time at which the data may still change (that is the reason we may not be able to respond immediately).Reads at times >=
upper
may not immediately return because the answer isn’t known yet. However, once theupper
is > the specified read time, the read can return.For the sake of returned values’ freshness, the coordinator prefers performing reads at an arrangement’s
upper
. However, because we more strongly prefer correctness, the coordinator will choose timestamps greater than an object’supper
if it is also being accessed alongside objects whosesince
times are >= itsupper
.
This illustration attempts to show, with time moving left to right, the
relationship between since
and upper
.
#
: possibly inaccurate results-
: immediate, correct response?
: not yet knowns
: sinceu
: upper|
: eligible for coordinator to select
####s----u?????
|||||||||||
Modules§
- appends
- Logic and types for all appends executed by the
Coordinator
. - catalog_
serving 🔒 - Special cases related to the “catalog serving” of Materialize
- caught_
up 🔒 - Support for checking whether clusters/collections are caught up during a 0dt deployment.
- cluster_
scheduling - command_
handler 🔒 - Logic for processing client
Command
s. EachCommand
is initiated by a client via some external Materialize API (ex: HTTP and psql). - consistency
- Internal consistency checks that validate invariants of
Coordinator
. - ddl 🔒
- This module encapsulates all of the
Coordinator
’s logic for creating, dropping, and altering objects. - id_
bundle 🔒 - in_
memory_ 🔒oracle - A timestamp oracle that relies on the
crate::catalog::Catalog
for persistence/durability and reserves ranges of timestamps. - indexes 🔒
- introspection 🔒
- Support for unified compute introspection.
- message_
handler 🔒 - Logic for processing
Coordinator
messages. TheCoordinator
receives messages from various sources (ex: controller, clients, background tasks, etc). - peek 🔒
- Logic and types for creating, executing, and tracking peeks.
- privatelink_
status 🔒 - read_
policy - Types and methods related to initializing, updating, and removing read policies on collections.
- sequencer 🔒
- Logic for executing a planned SQL query.
- sql 🔒
- Various utility methods used by the
Coordinator
. Ideally these are all put in more meaningfully named modules. - statement_
logging 🔒 - timeline 🔒
- A mechanism to ensure that a sequence of writes and reads proceed correctly through timestamps.
- timestamp_
selection 🔒 - Logic for selecting timestamps for various operations on collections.
- validity 🔒
Structs§
- Alter
Cluster - Alter
Cluster Finalize - Alter
Cluster Wait ForHydrated - Alter
Secret - Alter
Sink Ready Context - Background
Work Result - Cluster
Replica 🔒Statuses - Config
- Configures a coordinator.
- Conn
Meta - Metadata about an active connection.
- Coordinator
- Glues the external world to the Timely workers.
- Copy
ToContext - Create
Index Explain - Create
Index Finish - Create
Index Optimize - Create
Materialized View Explain - Create
Materialized View Finish - Create
Materialized View Optimize - Create
Secret Ensure - Create
Secret Finish - Create
View Explain - Create
View Finish - Create
View Optimize - Deferred
Plan 🔒Statement - Execute
Context - Bundle of state related to statement execution.
- Execute
Context Extra - State that the coordinator must process as part of retiring
command execution.
ExecuteContextExtra::Default
is guaranteed to produce a value that will cause the coordinator to do nothing, and is intended for use by code that invokes the execution processing flow (i.e.,sequence_plan
) without actually being a statement execution. - Execute
Context Inner - Explain
Plan Context - Explain
Timestamp Finish - Explain
Timestamp Optimize - Explain
Timestamp Real Time Recency - Introspection
Subscribe Finish - Introspection
Subscribe Optimize Mir - Introspection
Subscribe Timestamp Optimize Lir - Last
Message 🔒 - Contains information about the last message the
Coordinator
processed. - Locked
VecDeque 🔒 - A struct for tracking the ownership of a lock and a VecDeque to store to-be-done work after the lock is freed.
- Peek
Stage Copy To - Peek
Stage Explain Plan - Peek
Stage Explain Pushdown - Peek
Stage Finish - Peek
Stage Linearize Timestamp - Peek
Stage Optimize - Peek
Stage Real Time Recency - Peek
Stage Timestamp Read Hold - Pending
Read Txn - A pending read transaction waiting to be linearized along with metadata about it’s state
- Pending
Txn - A pending transaction waiting to be committed.
- Rotate
Keys Secret Ensure - Rotate
Keys Secret Finish - Subscribe
Finish - Subscribe
Optimize Mir - Subscribe
Timestamp Optimize Lir - Validation
Ready
Enums§
- Cluster
Stage - Controller
Readiness - The reason for why a controller needs processing on the main loop.
- Create
Index Stage - Create
Materialized View Stage - Create
View Stage - Explain
Context - Explain
Timestamp Stage - Introspection
Subscribe Stage - Message
- Network
Policy Error - Peek
Stage - Pending
Read 🔒 - A pending read transaction waiting to be linearized.
- Pending
TxnResponse - The response we’ll send for a
PendingTxn
. - Plan
Statement 🔒 - Secret
Stage - Stage
Result 🔒 - Result types for each stage of a sequence.
- Subscribe
Stage - Target
Cluster - An enum describing which cluster to run a statement on.
- Watch
SetResponse
Traits§
- Staged 🔒
- Common functionality for Coordinator::sequence_staged.
- Staged
Context
Functions§
- get_
initial_ 🔒oracle_ timestamps - load_
remote_ system_ parameters - serve
- Serves the coordinator based on the provided configuration.
- validate_
ip_ 🔒with_ policy_ rules