Function dangerous_force_compaction_and_break_pushdown

pub async fn dangerous_force_compaction_and_break_pushdown<K, V, T, D>(
    write: &WriteHandle<K, V, T, D>,
    fuel: impl Fn() -> usize,
    wait: impl Fn() -> Duration,
)
where
    K: Debug + Codec,
    V: Debug + Codec,
    T: Timestamp + Lattice + Codec64,
    D: Semigroup + Ord + Codec64 + Send + Sync,

Attempts to compact all batches in a shard into a minimal number.

This destroys most hope of filter pushdown doing anything useful. If you think you want this, you almost certainly want something else, please come talk to the persist team before using.

This is accomplished by adding artificial fuel (representing some count of updates) to the shard’s internal Spine of batches structure on some schedule and doing any compaction work that results. Both the amount of fuel and cadence are dynamically tunable. However, it is not clear exactly how much fuel is safe to add in each call: 10 definitely seems fine, usize::MAX may not be. This also adds to the danger in “dangerous”.

This is intentionally not even hooked up to persistcli admin for the above reasons.

Exits once the shard is sufficiently compacted, but can be called in a loop to keep it compacted.