Expand description

Translation of SQL commands into timestamped Controller commands.

The various SQL commands instruct the system to take actions that are not yet explicitly timestamped. On the other hand, the underlying data continually change as time moves forward. On the third hand, we greatly benefit from the information that some times are no longer of interest, so that we may compact the representation of the continually changing collections.

The Coordinator curates these interactions by observing the progress collections make through time, choosing timestamps for its own commands, and eventually communicating that certain times have irretrievably “passed”.

Frontiers another way

If the above description of frontiers left you with questions, this repackaged explanation might help.

  • since is the least recent time (i.e. oldest time) that you can read from sources and be guaranteed that the returned data is accurate as of that time.

    Reads at times less than since may return values that were not actually seen at the specified time, but arrived later (i.e. the results are compacted).

    For correctness’ sake, the coordinator never chooses to read at a time less than an arrangement’s since.

  • upper is the first time after the most recent time that you can read from sources and receive an immediate response. Alternately, it is the least time at which the data may still change (that is the reason we may not be able to respond immediately).

    Reads at times >= upper may not immediately return because the answer isn’t known yet. However, once the upper is > the specified read time, the read can return.

    For the sake of returned values’ freshness, the coordinator prefers performing reads at an arrangement’s upper. However, because we more strongly prefer correctness, the coordinator will choose timestamps greater than an object’s upper if it is also being accessed alongside objects whose since times are >= its upper.

This illustration attempts to show, with time moving left to right, the relationship between since and upper.

  • #: possibly inaccurate results
  • -: immediate, correct response
  • ?: not yet known
  • s: since
  • u: upper
  • |: eligible for coordinator to select
###s----u?????
    |||||||||||

Modules

Types and methods for building dataflow descriptions.

Logic and types for fast-path determination for dataflow execution.

indexes 🔒

Types and methods related to acquiring and releasing read holds on collections.

timeline 🔒

A mechanism to ensure that a sequence of writes and reads proceed correctly through timestamps.

Macros

Enforces critical section invariants for functions that perform writes to tables, e.g. INSERT, UPDATE.

Structs

Holds tables needing advancement.

State provided to a catalog transaction closure.

Configures a coordinator.

ConnMeta 🔒

Metadata about an active connection.

Glues the external world to the Timely workers.

This is the struct meant to be paired with Message::WriteLockGrant, but could theoretically be used to queue any deferred plan.

A pending write transaction that will be committing during the next group commit.

Information about the read capability requirements of a collection.

TxnReads 🔒

Timestamps used by writes in an Append command.

Enums

Deferred 🔒

An operation that is deferred while waiting for a lock.

The response from a Peek, with row multiplicities represented in unary.

Traits

Functions

Creates a description of the statement stmt.

Converts a Duration to a Timestamp representing the number of milliseconds contained in that Duration

Constructs an ExecuteResponse that that will send some rows to the client immediately, as opposed to asking the dataflow layer to send along the rows after some computation.

Serves the coordinator based on the provided configuration.