Module timely::dataflow::operators::core::capture::replay

Traits and types for replaying captured timely dataflow streams.

A type can be replayed into any timely dataflow scope if it presents as an iterator whose Item type implements EventIterator with the same timestamp. Other types can implement the ReplayInto trait, but this should be done with care, as there is a protocol the replayer follows that must be respected if the computation is to make sense.

Protocol

The stream of events produced by each EventIterator implementation must satisfy, starting from a default timestamp of Timestamp::minimum() with count 1,

1. The progress messages may only increment the count for a timestamp if the cumulative count for some prior or equal timestamp is positive.
2. The data messages map only use a timestamp if the cumulative count for some prior or equal timestamp is positive.

Alternately, the sequence of events should, starting from an initial count of 1 for the timestamp Default::default(), describe decrements to held capabilities or the production of capabilities in their future, or messages sent at times in the future of held capabilities.

The order is very important here. One can move Event::Message events arbitrarily earlier in the sequence, and Event::Progress events arbitrarily later, but one cannot move a Event::Progress message that discards a last capability before any Event::Message that would use that capability.

For an example, the Operate<T> implementation for capture::CaptureOperator<T, D, P> records exactly what data is presented at the operator, both in terms of progress messages and data received.

Notes

Provided no stream of events reports the consumption of capabilities it does not hold, any interleaving of the streams of events will still maintain the invariants above. This means that each timely dataflow replay operator can replay any number of streams, allowing the replay to occur in a timely dataflow computation with more or fewer workers than that in which the stream was captured.

Traits

• Replay
  Replay a capture stream into a scope with the same timestamp.