Skip to main content

Module rfc_2133_append_api

opendal_core::docs::rfcs

Module rfc_2133_append_api 

Source
Expand description

Append API

§Summary

Introduce append operations for OpenDAL which allow users to add data to a file.

§Motivation

OpenDAL has the write operation used to create a file and upload in parts. This is implemented based on multipart API. However, current approach has some limitations:

  • Data could be lost and not readable before w.close() returned Ok(())
  • File can’t be appended again after w.close() returned Ok(())

To address these issues, I propose adding an append operation. Users can create an appender that provides a reentrant append operation. Each append operation will add data to the end of the file, which can be read immediately after the operation.

§Guide-level explanation

The files created by the append operation can be appended via append API.

async fn append_test(op: Operation) -> Result<()> {
  // create writer
  let append = op.append("path_to_file").await?;

  let bs = read_from_file();
  append.append(bs).await?;

  let bs = read_from_another_file();
  append.append(bs).await?;

  // close the file
  append.close().await?;
}

Difference between the write and append operation:

  • write: Always create a new file, not readable until close.
  • append: Can append existing appendable file, readable after append return.

§Reference-level explanation

For underlay API, we will make these changes in Accessor:

trait Accessor {
    type Appender: oio::Append;

    async fn append(&self, path: &str, args: OpAppend) -> Result<Self::Append>;
}

To implement this feature, we need to add a new API append into oio::Append.

#[async_trait]
pub trait Append: Unpin + Send + Sync {
    /// Append data to the end of file.
    /// Users will call `append` multiple times. Please make sure `append` is safe to re-enter.
    async fn append(&mut self, bs: Bytes) -> Result<()>;

    /// Seal the file to mark it as unmodifiable.
    async fn close(&mut self) -> Result<()>;
}

§Drawbacks

None.

§Rationale and alternatives

None.

§Prior art

None.

§Unresolved questions

None.

§Future possibilities

We can use append API to implement for services that natively support append, such as Azure blob and Alibaba cloud OSS. This will improve the performance and reliability of append operation.