Module async_writer

Source
Expand description

async API for writing RecordBatches to Parquet files

See the crate-level documentation for more details.

The async API for writing RecordBatches is similar to the sync API, so please read the documentation there before using this API.

Here is an example for using AsyncArrowWriter:

let col = Arc::new(Int64Array::from_iter_values([1, 2, 3])) as ArrayRef;
let to_write = RecordBatch::try_from_iter([("col", col)]).unwrap();

let mut buffer = Vec::new();
let mut writer = AsyncArrowWriter::try_new(&mut buffer, to_write.schema(), None).unwrap();
writer.write(&to_write).await.unwrap();
writer.close().await.unwrap();

let buffer = Bytes::from(buffer);
let mut reader = ParquetRecordBatchReaderBuilder::try_new(buffer.clone())
    .unwrap()
    .build()
    .unwrap();
let read = reader.next().unwrap().unwrap();

assert_eq!(to_write, read);

[object_store] provides it’s native implementation of AsyncFileWriter by [ParquetObjectWriter].

Structs§

AsyncArrowWriter
Encodes RecordBatch to parquet, outputting to an AsyncFileWriter

Traits§

AsyncFileWriter
The asynchronous interface used by AsyncArrowWriter to write parquet files.