Skip to main content

compute_sampled_splits

Function compute_sampled_splits 

Source
async fn compute_sampled_splits<Q>(
    conn: &mut Q,
    table: &MySqlTableName,
    pk_col: &(String, SqlScalarType),
    worker_count: usize,
    total: u64,
) -> Result<Option<PkBoundaries>, TransientError>
where Q: Queryable,
Expand description

Walks the primary key index in steps of about row_count / worker_count, taking the key at each step’s OFFSET. The per-step OFFSET scans sum to a full index pass, so this function has a time complexity of O(row_count). Worker count is small, so the OFFSET scans dominate the runtime. row_count can be an optimizer estimate for large tables, so the partitions are approximate. An overestimate walks off the end of the index and stops with fewer boundaries, resulting in some workers receiving less or no work. An underestimate leaves a larger final partition for the last worker, however both still correctly partition the table. Returns None if the primary key column type is not supported or the table is too small to split.