Module operator

Expand description

Timely operators for the crate

Structs§

DataSubscribe: A helper for subscribing to a data shard using the timely operators.
DataSubscribeTask: A handle to a DataSubscribe running in a task.
TxnsContext: The process global TxnsRead that any operator can communicate with.

SourceEvent 🔒: Event sent from the subscribe Tokio task to the sync txns_progress_source operator. The task owns the persist resources and the data_subscribe receiver. The operator owns the output capability and drives the frontier.

txns_data_shard_retry_params: Retry configuration for txn-wal data shard override of next_listen_batch.
txns_progress: An operator for translating physical data shard frontiers into logical ones.
txns_progress_frontiers 🔒: The block ordering inside the schedule closure is load-bearing: pending passthrough input is emitted at the pre-activation capability BEFORE any capability downgrade, which keeps the differential invariant send_time <= record_time and avoids dropping in-flight rows when the passthrough frontier crosses until in the same activation (SQL-299). Do not reorder.
txns_progress_source_global 🔒: TODO: I’d much prefer the communication protocol between the two operators to be exactly remap as defined in the reclocking design doc. However, we can’t quite recover exactly the information necessary to construct that at the moment. Seems worth doing, but in the meantime, intentionally make this look fairly different (Stream of DataRemapEntry instead of Collection<FromTime>) to hopefully minimize confusion. As a performance optimization, we only re-emit this when the physical upper has changed, which means that the frontier of the Stream<DataRemapEntry<T>> indicates updates to the logical_upper of the most recent DataRemapEntry (i.e. the one with the largest physical_upper).