Struct Spine 

pub struct Spine<B: Batch> {

    operator: OperatorInfo,
    logger: Option<Logger>,
    logical_frontier: Antichain<B::Time>,
    physical_frontier: Antichain<B::Time>,
    merging: Vec<MergeState<B>>,
    pending: Vec<B>,
    upper: Antichain<B::Time>,
    effort: usize,
    activator: Option<Activator>,
    exert_logic_param: Vec<(usize, usize, usize)>,
    exert_logic: Option<ExertionLogic>,
    consolidation_credit: usize,
    progress_grants: usize,
}

An append-only collection of update tuples.

A spine maintains a small number of immutable collections of update tuples, merging the collections when two have similar sizes. In this way, it allows the addition of more tuples, which may then be merged with other immutable collections.

Fields

operator: OperatorInfo

logger: Option<Logger>

logical_frontier: Antichain<B::Time>

physical_frontier: Antichain<B::Time>

merging: Vec<MergeState<B>>

pending: Vec<B>

upper: Antichain<B::Time>

effort: usize

activator: Option<Activator>

exert_logic_param: Vec<(usize, usize, usize)>

Parameters to exert_logic, containing tuples of (index, count, length).

exert_logic: Option<ExertionLogic>

Logic to indicate whether and how many records we should introduce in the absence of actual updates.

consolidation_credit: usize

Fuel funded by inserted updates and not yet spent. It does not decay: it accumulates in proportion to total input, and later turns spend it down, so the total stays proportional to input.

progress_grants: usize

Policy allowances funded by inserted batches and not yet spent.

Implementations

impl<B: Batch> Spine<B>

fn drop_batches(&mut self)

Drops and logs batches. Used in set_logical_compaction and drop.

impl<B: Batch> Spine<B>

fn exert_effort(&mut self) -> Option<usize>

Determine the amount of effort we should exert in the absence of updates.

This method prepares an iterator over batches, including the level, count, and length of each layer. It supplies this to self.exert_logic, who produces the response of the amount of exertion to apply.

pub fn with_effort( effort: usize, operator: OperatorInfo, logger: Option<Logger>, activator: Option<Activator>, ) -> Self

Allocates a fueled Spine with a specified effort multiplier.

This trace will merge batches progressively, with each inserted batch applying a multiple of the batch’s length in effort to each merge. The effort parameter is that multiplier. This value should be at least one for the merging to happen; a value of zero is not helpful.

fn spend_funding(&mut self, effort: usize) -> bool

Spend funding for one policy request, returning whether any was available.

fn grant(&mut self, effort: usize)

Apply one policy allowance: fuel for active merges, or a virtual introduction.

fn consider_merges(&mut self)

Migrate data from self.pending into self.merging.

This method reflects on the bookmarks held by others that may prevent merging, and in the case that new batches can be introduced to the pile of mergeable batches, it gets on that.

pub fn introduce_batch(&mut self, batch: Option<B>, batch_index: usize)

Introduces a batch at an indicated level.

The level indication is often related to the size of the batch, but it can also be used to artificially fuel the computation by supplying empty batches at non-trivial indices, to move merges along.

fn roll_up(&mut self, index: usize)

Ensures that an insertion at layer index will succeed.

This method is subject to the constraint that all existing batches should occur at higher levels, which requires it to “roll up” batches present at lower levels before the method is called. In doing this, we should not introduce more virtual records than 2^index, as that is the amount of excess fuel we have budgeted for completing merges.

pub fn apply_fuel(&mut self, fuel: &mut isize)

Applies an amount of fuel to merges in progress.

The supplied fuel is for each in progress merge, and if we want to spend the fuel non-uniformly (e.g. prioritizing merges at low layers) we could do so in order to maintain fewer batches on average (at the risk of completing merges of large batches later, but tbh probably not much later).

fn insert_at(&mut self, batch: Option<B>, index: usize)

Inserts a batch at a specific location.

This is a non-public internal method that can panic if we try and insert into a layer which already contains two batches (and is still in the process of merging).

fn complete_at(&mut self, index: usize) -> Option<B>

Completes and extracts what ever is at layer index.

fn tidy_layers(&mut self)

Attempts to draw down large layers to size appropriate layers.

Trait Implementations

impl<B: Batch> Drop for Spine<B>

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl<B: Batch + Clone + 'static> Trace for Spine<B>

fn exert(&mut self)

Apply some amount of effort to trace maintenance.

Whether and how much effort to apply is determined by self.exert_logic, a closure the user can set.

fn close(&mut self)

Completes the trace with a final empty batch.

fn new( info: OperatorInfo, logging: Option<Logger>, activator: Option<Activator>, ) -> Self

Allocates a new empty trace.

fn set_exert_logic(&mut self, logic: ExertionLogic)

Sets the logic for exertion in the absence of updates.

fn insert(&mut self, batch: Self::Batch)

Introduces a batch of updates to the trace.

impl<B: Batch + Clone + 'static> TraceReader for Spine<B>

type Time = <B as BatchReader>::Time

The timestamp type of the trace’s updates.

type Batch = B

The type of an immutable collection of updates.

fn batches_through( &mut self, upper: AntichainRef<'_, Self::Time>, ) -> Option<Vec<Self::Batch>>

Acquires the non-empty sequence of batches covering updates at times not greater or equal to an element of upper.

fn set_logical_compaction(&mut self, frontier: AntichainRef<'_, B::Time>)

Advances the frontier that constrains logical compaction.

fn get_logical_compaction(&mut self) -> AntichainRef<'_, B::Time>

Reports the logical compaction frontier.

fn set_physical_compaction(&mut self, frontier: AntichainRef<'_, B::Time>)

Advances the frontier that constrains physical compaction.

fn get_physical_compaction(&mut self) -> AntichainRef<'_, B::Time>

Reports the physical compaction frontier.

fn map_batches<F: FnMut(&Self::Batch)>(&self, f: F)

Maps logic across the non-empty sequence of batches in the trace.

fn cursor( &mut self, ) -> (CursorList<<Self::Batch as Navigable>::Cursor>, Vec<Self::Batch>)

where Self::Batch: Navigable,

Provides a cursor over updates contained in the trace.

fn cursor_through( &mut self, upper: AntichainRef<'_, Self::Time>, ) -> Option<(CursorList<<Self::Batch as Navigable>::Cursor>, Vec<Self::Batch>)>

where Self::Batch: Navigable,

Acquires a cursor to the restriction of the collection’s contents to updates at times not greater or equal to an element of upper.

fn read_upper(&mut self, target: &mut Antichain<Self::Time>)

Reads the upper frontier of committed times.

fn advance_upper(&mut self, upper: &mut Antichain<Self::Time>)

Advances upper by any empty batches.