Module rfc_0337_dir_entry 

Dir entry

• Proposal Name: dir_entry
• Start Date: 2022-06-08
• RFC PR: apache/opendal#337
• Tracking Issue: apache/opendal#338

Summary

Returning DirEntry instead of Object in list.

Motivation

In Object Stream, we introduce read_dir support via:

pub trait ObjectStream: futures::Stream<Item = Result<Object>> + Unpin + Send {}
impl<T> ObjectStream for T where T: futures::Stream<Item = Result<Object>> + Unpin + Send {}

pub struct Object {
    acc: Arc<dyn Accessor>,
    meta: Metadata,
}

However, the meta inside Object is not well-used:

pub(crate) fn metadata_ref(&self) -> &Metadata {}
pub(crate) fn metadata_mut(&mut self) -> &mut Metadata {}
pub async fn metadata_cached(&mut self) -> Result<&Metadata> {}

Users can’t know an object’s mode after the list, so they have to send metadata every time they get an object:

let o = op.object("path/to/dir/");
let mut obs = o.list().await?;
// ObjectStream implements `futures::Stream`
while let Some(o) = obs.next().await {
    let mut o = o?;
    // It's highly possible that OpenDAL already did metadata during list.
    // Use `Object::metadata_cached()` to get cached metadata at first.
    let meta = o.metadata_cached().await?;
    match meta.mode() {
        ObjectMode::FILE => {
            println!("Handling file")
        }
        ObjectMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        ObjectMode::Unknown => continue,
    }
}

This behavior doesn’t make sense as we already know the object’s mode after the list.

Introducing a separate DirEntry could reduce an extra call for metadata most of the time.

let o = op.object("path/to/dir/");
let mut ds = o.list().await?;
// ObjectStream implements `futures::Stream`
while let Some(de) = ds.try_next().await {
    match de.mode() {
        ObjectMode::FILE => {
            println!("Handling file")
        }
        ObjectMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        ObjectMode::Unknown => continue,
    }
}

Guide-level explanation

Within this RFC, Object::list() will return DirStreamer instead.

pub trait DirStream: futures::Stream<Item = Result<DirEntry>> + Unpin + Send {}
pub type DirStreamer = Box<dyn DirStream>;

DirStreamer will stream DirEntry, which carries information already known during the list. So we can:

let id = de.id();
let path = de.path();
let name = de.name();
let mode = de.mode();
let meta = de.metadata().await?;

With DirEntry support, we can reduce an extra metadata call if we only want to know the object’s mode:

let o = op.object("path/to/dir/");
let mut ds = o.list().await?;
// ObjectStream implements `futures::Stream`
while let Some(de) = ds.try_next().await {
    match de.mode() {
        ObjectMode::FILE => {
            println!("Handling file")
        }
        ObjectMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        ObjectMode::Unknown => continue,
    }
}

We can convert this DirEntry into Object without overhead:

let o = de.into();

Reference-level explanation

This proposal will introduce a new struct, DirEntry:

struct DirEntry {}

impl DirEntry {
    pub fn id() -> String {}
    pub fn path() -> &str {}
    pub fn name() -> &str {}
    pub fn mode() -> ObjectMode {}
    pub async fn metadata() -> ObjectMetadata {}
}

impl From<DirEntry> for Object {}

And use DirStream to replace ObjectStream:

pub trait DirStream: futures::Stream<Item = Result<DirEntry>> + Unpin + Send {}
pub type DirStreamer = Box<dyn DirStream>;

With the addition of DirEntry, we will remove meta from Object:

#[derive(Clone, Debug)]
pub struct Object {
    acc: Arc<dyn Accessor>,
    path: String,
}

After this change, Object will become a thin wrapper of Accessor with path. And metadata related APIs like metadata_ref() and metadata_mut() will also be removed.

Drawbacks

We are adding a new concept to our core logic.

Rationale and alternatives

Rust fs API design

Rust also provides abstractions like File and DirEntry:

use std::fs;

fn main() -> std::io::Result<()> {
    for entry in fs::read_dir(".")? {
        let dir = entry?;
        println!("{:?}", dir.path());
    }
    Ok(())
}

Users can open a file with entry.path().

Prior art

None.

Unresolved questions

None.

Future possibilities

None.