parquet::arrow

Module async_writer

Source
Expand description

Contains async writer which writes arrow data into parquet data.

Provides async API for writing RecordBatches as parquet files. The API is similar to the sync API, so please read the documentation there before using this API.

Here is an example for using AsyncArrowWriter:

let col = Arc::new(Int64Array::from_iter_values([1, 2, 3])) as ArrayRef;
let to_write = RecordBatch::try_from_iter([("col", col)]).unwrap();

let mut buffer = Vec::new();
let mut writer = AsyncArrowWriter::try_new(&mut buffer, to_write.schema(), None).unwrap();
writer.write(&to_write).await.unwrap();
writer.close().await.unwrap();

let buffer = Bytes::from(buffer);
let mut reader = ParquetRecordBatchReaderBuilder::try_new(buffer.clone())
    .unwrap()
    .build()
    .unwrap();
let read = reader.next().unwrap().unwrap();

assert_eq!(to_write, read);

[object_store] provides it’s native implementation of AsyncFileWriter by [ParquetObjectWriter].

Structs§

Traits§