Constant mz_persist::indexed::columnar::KEY_VAL_DATA_MAX_LEN

pub const KEY_VAL_DATA_MAX_LEN: usize = _; // 2_147_483_647usize

The maximum allowed amount of total key data (similarly val data) in a single ColumnarBatch.

Note that somewhat counter-intuitively, this also includes offsets (counting as 4 bytes each) in the definition of “key/val data”.

TODO: The limit on the amount of {key,val} data is because we use i32 offsets in parquet; this won’t change. However, we include the offsets in the size because the parquet library we use currently maps each Array 1:1 with a parquet “page” (so for a “binary” column this is both the offsets and the data). The parquet format internally stores the size of a page in an i32, so if this gets too big, our library overflows it and writes bad data. There’s no reason it needs to map an Array 1:1 to a page (it could instead be 1:1 with a “column chunk”, which contains 1 or more pages). For now, we work around it.