Module mz_persist_types::arrow

A protobuf representation of Apache Arrow arrays.

Motivation

Persist can store a small amount of data inline at the consensus layer. Because we are space constrained, we take particular care to store only the data that is necessary. Other Arrow serialization formats, e.g. Parquet or Arrow IPC, include data that we don’t need and would be wasteful to store.

Modules

• proto 🔒

Structs

• ArrayBound
  An array with precisely one entry, for use as a lower bound.
• ArrayIdx
  A struct representing a particular entry in a particular array. Most useful for its Ord implementation, which can compare entire rows across similarly-typed arrays.
• ProtoArrayData
• ProtoDataType

Enums

• ArrayOrd
  Wraps a single arrow array, downcasted to a specific type.

Functions

• fields_for_type
  Extract the list of fields for our recursive datatypes.
• from_proto_with_type 🔒
  Decode the array data. If the data type is omitted from the proto, we decode it as the expected type.
• into_proto_with_type 🔒
  Encode the array into proto. If an expected data type is passed, that implies it is encoded at some higher level, and we omit it from the data.
• list_range 🔒
• maybe_trim_proto 🔒
  Makes a best effort to shrink the proto while preserving the ordering. (The proto might not be smaller after this method is called, but it should always be a valid lower bound.)