Struct parquet::arrow::arrow_writer::ArrowWriter

source ·
pub struct ArrowWriter<W: Write> { /* private fields */ }
Expand description

Encodes RecordBatch to parquet

Writes Arrow RecordBatches to a Parquet writer. Multiple RecordBatch will be encoded to the same row group, up to max_row_group_size rows. Any remaining rows will be flushed on close, leading the final row group in the output file to potentially contain fewer than max_row_group_size rows

let col = Arc::new(Int64Array::from_iter_values([1, 2, 3])) as ArrayRef;
let to_write = RecordBatch::try_from_iter([("col", col)]).unwrap();

let mut buffer = Vec::new();
let mut writer = ArrowWriter::try_new(&mut buffer, to_write.schema(), None).unwrap();
writer.write(&to_write).unwrap();
writer.close().unwrap();

let mut reader = ParquetRecordBatchReader::try_new(Bytes::from(buffer), 1024).unwrap();
let read = reader.next().unwrap().unwrap();

assert_eq!(to_write, read);

§Memory Limiting

The nature of parquet forces buffering of an entire row group before it can be flushed to the underlying writer. Data is buffered in its encoded form, to reduce memory usage, but if writing rows containing large strings or very nested data, this may still result in non-trivial memory usage.

ArrowWriter::in_progress_size can be used to track the size of the buffered row group, and potentially trigger an early flush of a row group based on a memory threshold and/or global memory pressure. However, users should be aware that smaller row groups will result in higher metadata overheads, and may worsen compression ratios and query performance.

writer.write(&batch).unwrap();
// Trigger an early flush if buffered size exceeds 1_000_000
if writer.in_progress_size() > 1_000_000 {
    writer.flush().unwrap();
}

Implementations§

source§

impl<W: Write + Send> ArrowWriter<W>

source

pub fn try_new( writer: W, arrow_schema: SchemaRef, props: Option<WriterProperties>, ) -> Result<Self>

Try to create a new Arrow writer

The writer will fail if:

  • a SerializedFileWriter cannot be created from the ParquetWriter
  • the Arrow schema contains unsupported datatypes such as Unions
source

pub fn try_new_with_options( writer: W, arrow_schema: SchemaRef, options: ArrowWriterOptions, ) -> Result<Self>

Try to create a new Arrow writer with ArrowWriterOptions.

The writer will fail if:

  • a SerializedFileWriter cannot be created from the ParquetWriter
  • the Arrow schema contains unsupported datatypes such as Unions
source

pub fn flushed_row_groups(&self) -> &[RowGroupMetaDataPtr]

Returns metadata for any flushed row groups

source

pub fn in_progress_size(&self) -> usize

Returns the estimated length in bytes of the current in progress row group

source

pub fn in_progress_rows(&self) -> usize

Returns the number of rows buffered in the in progress row group

source

pub fn bytes_written(&self) -> usize

Returns the number of bytes written by this instance

source

pub fn write(&mut self, batch: &RecordBatch) -> Result<()>

Encodes the provided RecordBatch

If this would cause the current row group to exceed WriterProperties::max_row_group_size rows, the contents of batch will be written to one or more row groups such that all but the final row group in the file contain WriterProperties::max_row_group_size rows.

This will fail if the batch’s schema does not match the writer’s schema.

source

pub fn flush(&mut self) -> Result<()>

Flushes all buffered rows into a new row group

source

pub fn append_key_value_metadata(&mut self, kv_metadata: KeyValue)

Additional KeyValue metadata to be written in addition to those from WriterProperties

This method provide a way to append kv_metadata after write RecordBatch

source

pub fn inner(&self) -> &W

Returns a reference to the underlying writer.

source

pub fn inner_mut(&mut self) -> &mut W

Returns a mutable reference to the underlying writer.

It is inadvisable to directly write to the underlying writer, doing so will likely result in a corrupt parquet file

source

pub fn into_inner(self) -> Result<W>

Flushes any outstanding data and returns the underlying writer.

source

pub fn finish(&mut self) -> Result<FileMetaData>

Close and finalize the underlying Parquet writer

Unlike Self::close this does not consume self

Attempting to write after calling finish will result in an error

source

pub fn close(self) -> Result<FileMetaData>

Close and finalize the underlying Parquet writer

Trait Implementations§

source§

impl<W: Write + Send> Debug for ArrowWriter<W>

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
source§

impl<W: Write + Send> RecordBatchWriter for ArrowWriter<W>

source§

fn write(&mut self, batch: &RecordBatch) -> Result<(), ArrowError>

Write a single batch to the writer.
source§

fn close(self) -> Result<(), ArrowError>

Write footer or termination data, then mark the writer as done.

Auto Trait Implementations§

§

impl<W> Freeze for ArrowWriter<W>
where W: Freeze,

§

impl<W> !RefUnwindSafe for ArrowWriter<W>

§

impl<W> Send for ArrowWriter<W>
where W: Send,

§

impl<W> !Sync for ArrowWriter<W>

§

impl<W> Unpin for ArrowWriter<W>
where W: Unpin,

§

impl<W> !UnwindSafe for ArrowWriter<W>

Blanket Implementations§

source§

impl<T> Any for T
where T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<T> Borrow<T> for T
where T: ?Sized,

source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<T> From<T> for T

source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T, U> Into<U> for T
where U: From<T>,

source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

§

type Error = Infallible

The type returned in the event of a conversion error.
source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

source§

fn vzip(self) -> V