Module operator

Expand description

Timely operators for the crate

Structs§

DataSubscribe: A helper for subscribing to a data shard using the timely operators.
DataSubscribeTask: A handle to a DataSubscribe running in a task.
TxnsContext: The process global TxnsRead that any operator can communicate with.
TxnsProgress: A subscription to one data shard’s remap information, from which any number of streams read off that shard can have their frontiers translated.

SourceEvent 🔒: Event sent from the subscribe Tokio task to the sync txns_progress_source operator. The task owns the persist resources and the data_subscribe receiver. The operator owns the output capability and drives the frontier.

txns_data_shard_retry_params: Retry configuration for txn-wal data shard override of next_listen_batch.
txns_progress: An operator for translating physical data shard frontiers into logical ones.
txns_progress_source_global 🔒: TODO: I’d much prefer the communication protocol between the two operators to be exactly remap as defined in the reclocking design doc. However, we can’t quite recover exactly the information necessary to construct that at the moment. Seems worth doing, but in the meantime, intentionally make this look fairly different (Stream of DataRemapEntry instead of Collection<FromTime>) to hopefully minimize confusion. As a performance optimization, we only re-emit this when the physical upper has changed, which means that the frontier of the Stream<DataRemapEntry<T>> indicates updates to the logical_upper of the most recent DataRemapEntry (i.e. the one with the largest physical_upper).