Module mz_persist::indexed::columnar::arrow

Apache Arrow encodings and utils for persist data

Constants

• ENABLE_ARROW_LGALLOC_CC_SIZES 🔒
• ENABLE_ARROW_LGALLOC_NONCC_SIZES 🔒

Statics

• SCHEMA_ARROW_RS_KVTD
  The Arrow schema we use to encode ((K, V), T, D) tuples.

Functions

• decode_arrow_batch_kvtd
  Converts an arrow [(K, V, T, D)] RecordBatch into a ColumnarRecords.
• decode_arrow_batch_kvtd_ks_vs
  Converts an arrow [(K, V, T, D)] Chunk into a ColumnarRecords.
• encode_arrow_batch_kvtd
  Converts a ColumnarRecords into (K, V, T, D) arrow columns.
• encode_arrow_batch_kvtd_ks_vs
  Converts a ColumnarRecords and ColumnarRecordsStructuredExt pair (aka BlobTraceUpdates::Both) into arrow::array::Arrays with columns [(K, V, T, D, K_S, V_S)].
• realloc_any
  Re-allocate the backing storage for an array ref using lgalloc, if it’s configured. (And hopefully-temporarily work around a parquet decoding issue upstream.)
• realloc_array
  Re-allocate the backing storage for a specific array using lgalloc, if it’s configured. (And hopefully-temporarily work around a parquet decoding issue upstream.)
• realloc_buffer 🔒
• realloc_data 🔒