Enum ProtoBatchFormat

#[repr(i32)]
pub enum ProtoBatchFormat {
    Unknown = 0,
    ArrowKvtd = 1,
    ParquetKvtd = 2,
    ParquetStructured = 3,
}

Variants

Unknown = 0

ArrowKvtd = 1

Arrow, which we’d use for the local blob cache if we use it, gets a structure like \[(K, V, T, D)\] so that we could mmap it and use it directly as our in-mem batches (which have this structure).

ParquetKvtd = 2

We have more flexibility with Parquet. Initially we’ll start with the same \[(K, V, T, D)\] as our in-mem batches. Another option would be something like \[(K, [(V, [(T, D)\])])], which would only store each key’s and each val’s data once (this is similar to the [differential_dataflow::trace::layers::Trie] structure of [differential_dataflow::trace::implementations::ord::OrdValBatch]).

Which is better probably comes down to how much duplication we expect of keys and vals in a batch as well as how big the batches are (the trie structure introduces more columns, so has some amount of overhead).

For unsealed batches, we have a better chance of duplicates than trace, but we probably don’t want to pay the cost of converting between the in-mem \[(K, V, T, D)\] representation and anything else (to keep the hot path clean). Unsealed batches are also likely to be our smallest. For this reason, they’ll probably always stay as ParquetKvtd.

For trace batches, we consolidate them before writing them out, so we’re guaranteed to get nothing from the V level of the trie. For duplicate keys, we’ll probably get a good amount of benefit from column specific compression, and I’d like to exhaust that direction first before dealing with a trie-like column structure.

ParquetStructured = 3

Parquet format that understands the structure of the inner data. See the metadata field on ProtoBatchPartInline for more information about how this data is structured in Parquet.

For example, the initial use of this format will contain the columns \[(K, V, T, D, K_S, V_S)\] where K_S and V_S are structured versions of the opaque data stored in K and V, respectively. Eventually we’ll stop writing K and V and migrate entirely to K_S and V_S.

Implementations

impl ProtoBatchFormat

pub fn is_valid(value: i32) -> bool

Returns true if value is a variant of ProtoBatchFormat.

pub fn from_i32(value: i32) -> Option<ProtoBatchFormat>

👎Deprecated: Use the TryFrom<i32> implementation instead

Converts an i32 to a ProtoBatchFormat, or None if value is not a valid variant.

impl ProtoBatchFormat

pub fn as_str_name(&self) -> &'static str

String value of the enum field names used in the ProtoBuf definition.

The values are not transformed in any way and thus are considered stable (if the ProtoBuf definition does not change) and safe for programmatic use.

pub fn from_str_name(value: &str) -> Option<Self>

Creates an enum from field names used in the ProtoBuf definition.

Trait Implementations

impl Clone for ProtoBatchFormat

fn clone(&self) -> ProtoBatchFormat

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ProtoBatchFormat

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ProtoBatchFormat

fn default() -> ProtoBatchFormat

Returns the “default value” for a type.

impl From<ProtoBatchFormat> for i32

fn from(value: ProtoBatchFormat) -> i32

Converts to this type from the input type.

impl Hash for ProtoBatchFormat

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for ProtoBatchFormat

fn cmp(&self, other: &ProtoBatchFormat) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl PartialEq for ProtoBatchFormat

fn eq(&self, other: &ProtoBatchFormat) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for ProtoBatchFormat

fn partial_cmp(&self, other: &ProtoBatchFormat) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl TryFrom<i32> for ProtoBatchFormat

type Error = UnknownEnumValue

The type returned in the event of a conversion error.

fn try_from(value: i32) -> Result<ProtoBatchFormat, UnknownEnumValue>

Performs the conversion.

impl Copy for ProtoBatchFormat

impl Eq for ProtoBatchFormat

impl StructuralPartialEq for ProtoBatchFormat