Struct Reader

Source
pub struct Reader { /* private fields */ }
Expand description

Reader is designed to read data from given path in an asynchronous manner.

§Usage

Reader provides multiple ways to read data from given reader.

Reader implements Clone so you can clone it and store in place where ever you want.

§Direct

Reader provides public API including Reader::read. You can use those APIs directly without extra copy.

use opendal::Operator;
use opendal::Result;

async fn test(op: Operator) -> Result<()> {
    let r = op.reader("path/to/file").await?;
    let bs = r.read(0..1024).await?;
    Ok(())
}

§Read like Stream

use anyhow::Result;
use bytes::Bytes;
use futures::TryStreamExt;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let s = op
        .reader("path/to/file")
        .await?
        .into_bytes_stream(1024..2048)
        .await?;
    let bs: Vec<Bytes> = s.try_collect().await?;
    Ok(())
}

§Read like AsyncRead and AsyncBufRead

use anyhow::Result;
use bytes::Bytes;
use futures::AsyncReadExt;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let mut r = op
        .reader("path/to/file")
        .await?
        .into_futures_async_read(1024..2048)
        .await?;
    let mut bs = vec![];
    let n = r.read_to_end(&mut bs).await?;
    Ok(())
}

Implementations§

Source§

impl Reader

Source

pub async fn read(&self, range: impl RangeBounds<u64>) -> Result<Buffer>

Read give range from reader into Buffer.

This operation is zero-copy, which means it keeps the bytes::Bytes returned by underlying storage services without any extra copy or intensive memory allocations.

Source

pub async fn read_into( &self, buf: &mut impl BufMut, range: impl RangeBounds<u64>, ) -> Result<usize>

Read all data from reader into given BufMut.

This operation will copy and write bytes into given BufMut. Allocation happens while BufMut doesn’t have enough space.

Source

pub async fn fetch(&self, ranges: Vec<Range<u64>>) -> Result<Vec<Buffer>>

Fetch specific ranges from reader.

This operation try to merge given ranges into a list of non-overlapping ranges. Users may also specify a gap to merge close ranges.

The returning Buffer may share the same underlying memory without any extra copy.

Source

pub async fn into_stream( self, range: impl RangeBounds<u64>, ) -> Result<BufferStream>

Create a buffer stream to read specific range from given reader.

§Notes

BufferStream is a zero-cost abstraction. It doesn’t involve extra copy of data. It will return underlying Buffer directly.

The Buffer this stream yields can be seen as an iterator of [Bytes].

§Inputs
  • range: The range of data to read. range like .. it will read all data from reader.
§Examples
§Basic Usage
use std::io;

use bytes::Bytes;
use futures::TryStreamExt;
use opendal::Buffer;
use opendal::Operator;
use opendal::Result;

async fn test(op: Operator) -> io::Result<()> {
    let mut s = op
        .reader("hello.txt")
        .await?
        .into_stream(1024..2048)
        .await?;

    let bs: Vec<Buffer> = s.try_collect().await?;
    // We can use those buffer as bytes if we want.
    let bytes_vec: Vec<Bytes> = bs.clone().into_iter().flatten().collect();
    // Or we can merge them into a single [`Buffer`] and later use it as [`bytes::Buf`].
    let new_buffer: Buffer = bs.into_iter().flatten().collect::<Buffer>();

    Ok(())
}
§Concurrent Read

The following example reads data in 256B chunks with 8 concurrent.

use std::io;

use bytes::Bytes;
use futures::TryStreamExt;
use opendal::Buffer;
use opendal::Operator;
use opendal::Result;

async fn test(op: Operator) -> io::Result<()> {
    let s = op
        .reader_with("hello.txt")
        .concurrent(8)
        .chunk(256)
        .await?
        .into_stream(1024..2048)
        .await?;

    // Every buffer except the last one in the stream will be 256B.
    let bs: Vec<Buffer> = s.try_collect().await?;
    // We can use those buffer as bytes if we want.
    let bytes_vec: Vec<Bytes> = bs.clone().into_iter().flatten().collect();
    // Or we can merge them into a single [`Buffer`] and later use it as [`bytes::Buf`].
    let new_buffer: Buffer = bs.into_iter().flatten().collect::<Buffer>();

    Ok(())
}
Source

pub async fn into_futures_async_read( self, range: impl RangeBounds<u64>, ) -> Result<FuturesAsyncReader>

Convert reader into FuturesAsyncReader which implements futures::AsyncRead, futures::AsyncSeek and futures::AsyncBufRead.

§Notes

FuturesAsyncReader is not a zero-cost abstraction. The underlying reader returns an owned Buffer, which involves an extra copy operation.

§Inputs
  • range: The range of data to read. range like .. it will read all data from reader.
§Examples
§Basic Usage
use std::io;

use futures::io::AsyncReadExt;
use opendal::Operator;
use opendal::Result;

async fn test(op: Operator) -> io::Result<()> {
    let mut r = op
        .reader("hello.txt")
        .await?
        .into_futures_async_read(1024..2048)
        .await?;
    let mut bs = Vec::new();
    r.read_to_end(&mut bs).await?;

    Ok(())
}
§Concurrent Read

The following example reads data in 256B chunks with 8 concurrent.

use std::io;

use futures::io::AsyncReadExt;
use opendal::Operator;
use opendal::Result;

async fn test(op: Operator) -> io::Result<()> {
    let mut r = op
        .reader_with("hello.txt")
        .concurrent(8)
        .chunk(256)
        .await?
        .into_futures_async_read(1024..2048)
        .await?;
    let mut bs = Vec::new();
    r.read_to_end(&mut bs).await?;

    Ok(())
}
Source

pub async fn into_bytes_stream( self, range: impl RangeBounds<u64>, ) -> Result<FuturesBytesStream>

Convert reader into FuturesBytesStream which implements futures::Stream.

§Inputs
  • range: The range of data to read. range like .. it will read all data from reader.
§Examples
§Basic Usage
use std::io;

use bytes::Bytes;
use futures::TryStreamExt;
use opendal::Operator;
use opendal::Result;

async fn test(op: Operator) -> io::Result<()> {
    let mut s = op
        .reader("hello.txt")
        .await?
        .into_bytes_stream(1024..2048)
        .await?;
    let bs: Vec<Bytes> = s.try_collect().await?;

    Ok(())
}
§Concurrent Read

The following example reads data in 256B chunks with 8 concurrent.

use std::io;

use bytes::Bytes;
use futures::TryStreamExt;
use opendal::Operator;
use opendal::Result;

async fn test(op: Operator) -> io::Result<()> {
    let mut s = op
        .reader_with("hello.txt")
        .concurrent(8)
        .chunk(256)
        .await?
        .into_bytes_stream(1024..2048)
        .await?;
    let bs: Vec<Bytes> = s.try_collect().await?;

    Ok(())
}

Trait Implementations§

Source§

impl Clone for Reader

Source§

fn clone(&self) -> Reader

Returns a duplicate of the value. Read more
1.0.0 · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more

Auto Trait Implementations§

§

impl Freeze for Reader

§

impl !RefUnwindSafe for Reader

§

impl Send for Reader

§

impl Sync for Reader

§

impl Unpin for Reader

§

impl !UnwindSafe for Reader

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T> Instrument for T

Source§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
Source§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> PolicyExt for T
where T: ?Sized,

Source§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns Action::Follow only if self and other return Action::Follow. Read more
Source§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns Action::Follow if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T> ServiceExt for T

Source§

fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>
where Self: Sized,

Apply a transformation to the response body. Read more
Source§

fn decompression(self) -> Decompression<Self>
where Self: Sized,

Decompress response bodies. Read more
Source§

fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using HTTP status codes. Read more
Source§

fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using gRPC headers. Read more
Source§

fn follow_redirects(self) -> FollowRedirect<Self>
where Self: Sized,

Follow redirect resposes using the Standard policy. Read more
Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V

Source§

impl<T> WithSubscriber for T

Source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

impl<T> MaybeSend for T
where T: Send,