Enum parquet::basic::Encoding

source ·
pub enum Encoding {
    PLAIN,
    PLAIN_DICTIONARY,
    RLE,
    BIT_PACKED,
    DELTA_BINARY_PACKED,
    DELTA_LENGTH_BYTE_ARRAY,
    DELTA_BYTE_ARRAY,
    RLE_DICTIONARY,
    BYTE_STREAM_SPLIT,
}
Expand description

Encodings supported by Parquet.

Not all encodings are valid for all types. These enums are also used to specify the encoding of definition and repetition levels.

By default this crate uses Encoding::PLAIN, Encoding::RLE, and Encoding::RLE_DICTIONARY. These provide very good encode and decode performance, whilst yielding reasonable storage efficiency and being supported by all major parquet readers.

The delta encodings are also supported and will be used if a newer WriterVersion is configured, however, it should be noted that these sacrifice encode and decode performance for improved storage efficiency. This performance regression is particularly pronounced in the case of record skipping as occurs during predicate push-down. It is recommended users assess the performance impact when evaluating these encodings.

Variants§

§

PLAIN

Default byte encoding.

  • BOOLEAN - 1 bit per value, 0 is false; 1 is true.
  • INT32 - 4 bytes per value, stored as little-endian.
  • INT64 - 8 bytes per value, stored as little-endian.
  • FLOAT - 4 bytes per value, stored as little-endian.
  • DOUBLE - 8 bytes per value, stored as little-endian.
  • BYTE_ARRAY - 4 byte length stored as little endian, followed by bytes.
  • FIXED_LEN_BYTE_ARRAY - just the bytes are stored.
§

PLAIN_DICTIONARY

Deprecated dictionary encoding.

The values in the dictionary are encoded using PLAIN encoding. Since it is deprecated, RLE_DICTIONARY encoding is used for a data page, and PLAIN encoding is used for dictionary page.

§

RLE

Group packed run length encoding.

Usable for definition/repetition levels encoding and boolean values.

§

BIT_PACKED

👎Deprecated: Please see documentation for compatibility issues and use the RLE/bit-packing hybrid encoding instead

Deprecated Bit-packed encoding.

This can only be used if the data has a known max width. Usable for definition/repetition levels encoding.

There are compatibility issues with files using this encoding. The parquet standard specifies the bits to be packed starting from the most-significant bit, several implementations do not follow this bit order. Several other implementations also have issues reading this encoding because of incorrect assumptions about the length of the encoded data.

The RLE/bit-packing hybrid is more cpu and memory efficient and should be used instead.

§

DELTA_BINARY_PACKED

Delta encoding for integers, either INT32 or INT64.

Works best on sorted data.

§

DELTA_LENGTH_BYTE_ARRAY

Encoding for byte arrays to separate the length values and the data.

The lengths are encoded using DELTA_BINARY_PACKED encoding.

§

DELTA_BYTE_ARRAY

Incremental encoding for byte arrays.

Prefix lengths are encoded using DELTA_BINARY_PACKED encoding. Suffixes are stored using DELTA_LENGTH_BYTE_ARRAY encoding.

§

RLE_DICTIONARY

Dictionary encoding.

The ids are encoded using the RLE encoding.

§

BYTE_STREAM_SPLIT

Encoding for floating-point data.

K byte-streams are created where K is the size in bytes of the data type. The individual bytes of an FP value are scattered to the corresponding stream and the streams are concatenated. This itself does not reduce the size of the data but can lead to better compression afterwards.

Trait Implementations§

source§

impl Clone for Encoding

source§

fn clone(&self) -> Encoding

Returns a copy of the value. Read more
1.0.0 · source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
source§

impl Debug for Encoding

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
source§

impl Display for Encoding

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
source§

impl From<Encoding> for Encoding

source§

fn from(value: Encoding) -> Self

Converts to this type from the input type.
source§

impl FromStr for Encoding

§

type Err = ParquetError

The associated error which can be returned from parsing.
source§

fn from_str(s: &str) -> Result<Self, Self::Err>

Parses a string s to return a value of this type. Read more
source§

impl Hash for Encoding

source§

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher. Read more
1.3.0 · source§

fn hash_slice<H>(data: &[Self], state: &mut H)
where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher. Read more
source§

impl Ord for Encoding

source§

fn cmp(&self, other: &Encoding) -> Ordering

This method returns an Ordering between self and other. Read more
1.21.0 · source§

fn max(self, other: Self) -> Self
where Self: Sized,

Compares and returns the maximum of two values. Read more
1.21.0 · source§

fn min(self, other: Self) -> Self
where Self: Sized,

Compares and returns the minimum of two values. Read more
1.50.0 · source§

fn clamp(self, min: Self, max: Self) -> Self
where Self: Sized + PartialOrd,

Restrict a value to a certain interval. Read more
source§

impl PartialEq for Encoding

source§

fn eq(&self, other: &Encoding) -> bool

This method tests for self and other values to be equal, and is used by ==.
1.0.0 · source§

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.
source§

impl PartialOrd for Encoding

source§

fn partial_cmp(&self, other: &Encoding) -> Option<Ordering>

This method returns an ordering between self and other values if one exists. Read more
1.0.0 · source§

fn lt(&self, other: &Rhs) -> bool

This method tests less than (for self and other) and is used by the < operator. Read more
1.0.0 · source§

fn le(&self, other: &Rhs) -> bool

This method tests less than or equal to (for self and other) and is used by the <= operator. Read more
1.0.0 · source§

fn gt(&self, other: &Rhs) -> bool

This method tests greater than (for self and other) and is used by the > operator. Read more
1.0.0 · source§

fn ge(&self, other: &Rhs) -> bool

This method tests greater than or equal to (for self and other) and is used by the >= operator. Read more
source§

impl TryFrom<Encoding> for Encoding

§

type Error = ParquetError

The type returned in the event of a conversion error.
source§

fn try_from(value: Encoding) -> Result<Self>

Performs the conversion.
source§

impl Copy for Encoding

source§

impl Eq for Encoding

source§

impl StructuralPartialEq for Encoding

Auto Trait Implementations§

Blanket Implementations§

source§

impl<T> Any for T
where T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<T> Borrow<T> for T
where T: ?Sized,

source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<Q, K> Equivalent<K> for Q
where Q: Eq + ?Sized, K: Borrow<Q> + ?Sized,

source§

fn equivalent(&self, key: &K) -> bool

Checks if this value is equivalent to the given key. Read more
source§

impl<T> From<T> for T

source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T, U> Into<U> for T
where U: From<T>,

source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T> ToOwned for T
where T: Clone,

§

type Owned = T

The resulting type after obtaining ownership.
source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
source§

impl<T> ToString for T
where T: Display + ?Sized,

source§

default fn to_string(&self) -> String

Converts the given value to a String. Read more
source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

§

type Error = Infallible

The type returned in the event of a conversion error.
source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

source§

fn vzip(self) -> V

source§

impl<T> Allocation for T
where T: RefUnwindSafe + Send + Sync,