Skip to main content

backpressure

Function backpressure 

Source
pub fn backpressure<'scope, T, O>(
    scope: Scope<'scope, (T, Subtime)>,
    name: &str,
    data: StreamVec<'scope, (T, Subtime), O>,
    flow_control: FlowControl<'scope, (T, Subtime)>,
    chosen_worker: usize,
    probe: Option<UnboundedSender<(Antichain<(T, Subtime)>, usize, usize)>>,
) -> (StreamVec<'scope, (T, Subtime), O>, PressOnDropButton)
Expand description

Apply flow control to the data input, based on the given FlowControl.

The FlowControl should have a progress_stream that is the pristine, unaltered frontier of the downstream operator we want to backpressure from, a max_inflight_bytes, and a summary. Note that the data input expects all the second part of the tuple timestamp to be 0, and all data to be on the chosen_worker worker.

The summary represents the minimum range of timestamps that needs to be emitted before reasoning about max_inflight_bytes. In practice this means that we may overshoot max_inflight_bytes.

The implementation of this operator is very subtle. Many inline comments have been added.