Module async_writer

async API for writing RecordBatches to Parquet files

See the crate-level documentation for more details.

The async API for writing RecordBatches is similar to the sync API, so please read the documentation there before using this API.

Here is an example for using AsyncArrowWriter:

let col = Arc::new(Int64Array::from_iter_values([1, 2, 3])) as ArrayRef;
let to_write = RecordBatch::try_from_iter([("col", col)]).unwrap();

let mut buffer = Vec::new();
let mut writer = AsyncArrowWriter::try_new(&mut buffer, to_write.schema(), None).unwrap();
writer.write(&to_write).await.unwrap();
writer.close().await.unwrap();

let buffer = Bytes::from(buffer);
let mut reader = ParquetRecordBatchReaderBuilder::try_new(buffer.clone())
    .unwrap()
    .build()
    .unwrap();
let read = reader.next().unwrap().unwrap();

assert_eq!(to_write, read);

[object_store] provides it’s native implementation of AsyncFileWriter by [ParquetObjectWriter].

Structs

AsyncArrowWriter

Encodes RecordBatch to parquet, outputting to an AsyncFileWriter

Traits

AsyncFileWriter

The asynchronous interface used by AsyncArrowWriter to write parquet files.