pub struct SizeStatistics {
pub unencoded_byte_array_data_bytes: Option<i64>,
pub repetition_level_histogram: Option<Vec<i64>>,
pub definition_level_histogram: Option<Vec<i64>>,
}
Expand description
A structure for capturing metadata for estimating the unencoded, uncompressed size of data written. This is useful for readers to estimate how much memory is needed to reconstruct data in their memory model and for fine grained filter pushdown on nested structures (the histograms contained in this structure can help determine the number of nulls at a particular nesting level and maximum length of lists).
Fields§
§unencoded_byte_array_data_bytes: Option<i64>
The number of physical bytes stored for BYTE_ARRAY data values assuming
no encoding. This is exclusive of the bytes needed to store the length of
each byte array. In other words, this field is equivalent to the (size of PLAIN-ENCODING the byte array values) - (4 bytes * number of values written)
. To determine unencoded sizes of other types readers can use
schema information multiplied by the number of non-null and null values.
The number of null/non-null values can be inferred from the histograms
below.
For example, if a column chunk is dictionary-encoded with dictionary [“a”, “bc”, “cde”], and a data page contains the indices [0, 0, 1, 2], then this value for that data page should be 7 (1 + 1 + 2 + 3).
This field should only be set for types that use BYTE_ARRAY as their physical type.
repetition_level_histogram: Option<Vec<i64>>
When present, there is expected to be one element corresponding to each repetition (i.e. size=max repetition_level+1) where each element represents the number of times the repetition level was observed in the data.
This field may be omitted if max_repetition_level is 0 without loss of information.
definition_level_histogram: Option<Vec<i64>>
Same as repetition_level_histogram except for definition levels.
This field may be omitted if max_definition_level is 0 or 1 without loss of information.
Implementations§
Trait Implementations§
Source§impl Clone for SizeStatistics
impl Clone for SizeStatistics
Source§fn clone(&self) -> SizeStatistics
fn clone(&self) -> SizeStatistics
1.0.0 · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read more