Module coord::coord

Coordination of installed views, available timestamps, compacted timestamps, and transactions.

The command coordinator maintains a view of the installed views, and for each tracks the frontier of available times (upper) and the frontier of compacted times (since). The upper frontier describes times that may not return immediately, as any timestamps in advance of the frontier are still open. The since frontier constrains those times for which the maintained view will be correct, as any timestamps in advance of the frontier must accumulate to the same value as would an un-compacted trace. The since frontier cannot be directly mutated, but instead can have multiple handles to it which forward changes from an internal MutableAntichain to the since.

The Coordinator tracks various compaction frontiers so that source, indexes, compaction, and transactions can work together. determine_timestamp() returns the least valid since of its sources. Any new transactions should thus always be >= the current compaction frontier and so should never change the frontier when being added to txn_reads. The compaction frontier may change when a transaction ends (if it was the oldest transaction and the since was advanced after the transaction started) or when update_upper() is run (if there are no in progress transactions before the new since). When it does, it is added to index_since_updates or source_since_updates and will be processed during the next maintenance() call.

Frontiers another way

If the above description of frontiers left you with questions, this repackaged explanation might help.

• since is the least recent time (i.e. oldest time) that you can read from sources and be guaranteed that the returned data is accurate as of that time.

  Reads at times less than since may return values that were not actually seen at the specified time, but arrived later (i.e. the results are compacted).

  For correctness’ sake, the coordinator never chooses to read at a time less than an arrangement’s since.

• upper is the first time after the most recent time that you can read from sources and receive an immediate response. Alternately, it is the least time at which the data may still change (that is the reason we may not be able to respond immediately).

  Reads at times >= upper may not immediately return because the answer isn’t known yet. However, once the upper is > the specified read time, the read can return.

  For the sake of returned values’ freshness, the coordinator prefers performing reads at an arrangement’s upper. However, because we more strongly prefer correctness, the coordinator will choose timestamps greater than an object’s upper if it is also being accessed alongside objects whose since times are >= its upper.

This illustration attempts to show, with time moving left to right, the relationship between since and upper.

• #: possibly inaccurate results
• -: immediate, correct response
• ?: not yet known
• s: since
• u: upper
• |: eligible for coordinator to select

###s----u?????
    |||||||||||

Modules

antichain

Antichain utilities.

arrangement_state

Frontier state for each arrangement.

dataflow_builder

Types and methods for building dataflow descriptions.

fast_path_peek

Logic and types for fast-path determination for dataflow execution.

prometheus

A helper that scrapes materialized’s prometheus metrics and produces updates for the mz_metrics table.

Macros

guard_write_critical_section

Enforces critical section invariants for functions that perform writes to tables, e.g. INSERT, UPDATE.

Structs

Config

Configures a coordinator.

ConnMeta

Metadata about an active connection.

Coordinator

Glues the external world to the Timely workers.

DeferredPlan

This is the struct meant to be paired with Message::WriteLockGrant, but could theoretically be used to queue any deferred plan.

LoggingConfig

Configures dataflow worker logging.

SendDiffs

SinkConnectorReady

StatementReady

TimestampedUpdate

TxnReads

Enums

ExprPrepStyle

The styles in which an expression can be prepared.

Message

Functions

auto_generate_primary_idx

check_statement_safety

describe

Creates a description of the statement stmt.

duration_to_timestamp_millis

Converts a Duration to a Timestamp representing the number of milliseconds contained in that Duration

index_sql

send_immediate_rows

Constructs an ExecuteResponse that that will send some rows to the client immediately, as opposed to asking the dataflow layer to send along the rows after some computation.

serve

Serves the coordinator based on the provided configuration.