Trait LimitProgress

trait LimitProgress<T: Timestamp> {
    // Required method
    fn limit_progress(
        &self,
        handle: MzProbeHandle<T>,
        slack_ms: u64,
        limit: Option<usize>,
        upper: Antichain<T>,
        name: String,
    ) -> (Rc<dyn Any>, Self);
}

Extension trait for StreamCore to selectively limit progress.

Required Methods

fn limit_progress( &self, handle: MzProbeHandle<T>, slack_ms: u64, limit: Option<usize>, upper: Antichain<T>, name: String, ) -> (Rc<dyn Any>, Self)

Limit the progress of the stream until its frontier reaches the given upper bound. Expects the implementation to observe times in data, and release capabilities based on the probe’s frontier, after applying slack to round up timestamps.

The implementation of this operator is subtle to avoid regressions in the rest of the system. Specifically joins hold back compaction on the other side of the join, so we need to make sure we release capabilities as soon as possible. This is why we only limit progress for times before the upper, which is the time until which the source can distinguish updates at the time of rendering. Once we make progress to the upper, we need to release our capability.

This isn’t perfect, and can result in regressions if on of the inputs lags behind. We could consider using the join of the uppers, i.e, use lower bound upper of all available inputs.

Once the input frontier reaches [], the implementation must release any capability to allow downstream operators to release resources.

The implementation should limit the number of pending times to limit if it is Some to avoid unbounded memory usage.

• handle is a probe installed on the dataflow’s outputs as late as possible, but before any timestamp rounding happens (c.f., REFRESH EVERY materialized views).
• slack_ms is the number of milliseconds to round up timestamps to.
• name is a human-readable name for the operator.
• limit is the maximum number of pending times to keep around.
• upper is the upper bound of the stream’s frontier until which the implementation can retain a capability.

Returns a shutdown button and the stream with the progress limiting applied.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementations on Foreign Types

impl<G, D, R> LimitProgress<Timestamp> for StreamCore<G, Vec<(D, Timestamp, R)>>

where G: Scope<Timestamp = Timestamp>, D: Data, R: Data,

fn limit_progress( &self, handle: MzProbeHandle<Timestamp>, slack_ms: u64, limit: Option<usize>, upper: Antichain<Timestamp>, name: String, ) -> (Rc<dyn Any>, Self)

Implementors