Module tokio::fs

source ·
Expand description

Asynchronous file utilities.

This module contains utility methods for working with the file system asynchronously. This includes reading/writing to files, and working with directories.

Be aware that most operating systems do not provide asynchronous file system APIs. Because of that, Tokio will use ordinary blocking file operations behind the scenes. This is done using the spawn_blocking threadpool to run them in the background.

The tokio::fs module should only be used for ordinary files. Trying to use it with e.g., a named pipe on Linux can result in surprising behavior, such as hangs during runtime shutdown. For special files, you should use a dedicated type such as tokio::net::unix::pipe or AsyncFd instead.

Currently, Tokio will always use spawn_blocking on all platforms, but it may be changed to use asynchronous file system APIs such as io_uring in the future.

§Usage

The easiest way to use this module is to use the utility functions that operate on entire files:

The two read functions reads the entire file and returns its contents. The write function takes the contents of the file and writes those contents to the file. It overwrites the existing file, if any.

For example, to read the file:

let contents = tokio::fs::read_to_string("my_file.txt").await?;

println!("File has {} lines.", contents.lines().count());

To overwrite the file:

let contents = "First line.\nSecond line.\nThird line.\n";

tokio::fs::write("my_file.txt", contents.as_bytes()).await?;

§Using File

The main type for interacting with files is File. It can be used to read from and write to a given file. This is done using the AsyncRead and AsyncWrite traits. This type is generally used when you want to do something more complex than just reading or writing the entire contents in one go.

Note: It is important to use flush when writing to a Tokio File. This is because calls to write will return before the write has finished, and flush will wait for the write to finish. (The write will happen even if you don’t flush; it will just happen later.) This is different from std::fs::File, and is due to the fact that File uses spawn_blocking behind the scenes.

For example, to count the number of lines in a file without loading the entire file into memory:

use tokio::fs::File;
use tokio::io::AsyncReadExt;

let mut file = File::open("my_file.txt").await?;

let mut chunk = vec![0; 4096];
let mut number_of_lines = 0;
loop {
    let len = file.read(&mut chunk).await?;
    if len == 0 {
        // Length of zero means end of file.
        break;
    }
    for &b in &chunk[..len] {
        if b == b'\n' {
            number_of_lines += 1;
        }
    }
}

println!("File has {} lines.", number_of_lines);

For example, to write a file line-by-line:

use tokio::fs::File;
use tokio::io::AsyncWriteExt;

let mut file = File::create("my_file.txt").await?;

file.write_all(b"First line.\n").await?;
file.write_all(b"Second line.\n").await?;
file.write_all(b"Third line.\n").await?;

// Remember to call `flush` after writing!
file.flush().await?;

§Tuning your file IO

Tokio’s file uses spawn_blocking behind the scenes, and this has serious performance consequences. To get good performance with file IO on Tokio, it is recommended to batch your operations into as few spawn_blocking calls as possible.

One example of this difference can be seen by comparing the two reading examples above. The first example uses tokio::fs::read, which reads the entire file in a single spawn_blocking call, and then returns it. The second example will read the file in chunks using many spawn_blocking calls. This means that the second example will most likely be more expensive for large files. (Of course, using chunks may be necessary for very large files that don’t fit in memory.)

The following examples will show some strategies for this:

When creating a file, write the data to a String or Vec<u8> and then write the entire file in a single spawn_blocking call with tokio::fs::write.

let mut contents = String::new();

contents.push_str("First line.\n");
contents.push_str("Second line.\n");
contents.push_str("Third line.\n");

tokio::fs::write("my_file.txt", contents.as_bytes()).await?;

Use BufReader and BufWriter to buffer many small reads or writes into a few large ones. This example will most likely only perform one spawn_blocking call.

use tokio::fs::File;
use tokio::io::{AsyncWriteExt, BufWriter};

let mut file = BufWriter::new(File::create("my_file.txt").await?);

file.write_all(b"First line.\n").await?;
file.write_all(b"Second line.\n").await?;
file.write_all(b"Third line.\n").await?;

// Due to the BufWriter, the actual write and spawn_blocking
// call happens when you flush.
file.flush().await?;

Manually use std::fs inside spawn_blocking.

use std::fs::File;
use std::io::{self, Write};
use tokio::task::spawn_blocking;

spawn_blocking(move || {
    let mut file = File::create("my_file.txt")?;

    file.write_all(b"First line.\n")?;
    file.write_all(b"Second line.\n")?;
    file.write_all(b"Third line.\n")?;

    // Unlike Tokio's file, the std::fs file does
    // not need flush.

    io::Result::Ok(())
}).await.unwrap()?;

It’s also good to be aware of File::set_max_buf_size, which controls the maximum amount of bytes that Tokio’s File will read or write in a single spawn_blocking call. The default is two megabytes, but this is subject to change.

Structs§

  • A builder for creating directories in various manners.
  • Entries returned by the ReadDir stream.
  • A reference to an open file on the filesystem.
  • Options and flags which can be used to configure how a file is opened.
  • Reads the entries in a directory.

Functions§

  • Returns the canonical, absolute form of a path with all intermediate components normalized and symbolic links resolved.
  • Copies the contents of one file to another. This function will also copy the permission bits of the original file to the destination file. This function will overwrite the contents of to.
  • Creates a new, empty directory at the provided path.
  • Recursively creates a directory and all of its parent components if they are missing.
  • Creates a new hard link on the filesystem.
  • Given a path, queries the file system to get information about a file, directory, etc.
  • Reads the entire contents of a file into a bytes vector.
  • Returns a stream over the entries within a directory.
  • Reads a symbolic link, returning the file that the link points to.
  • Creates a future which will open a file for reading and read the entire contents into a string and return said string.
  • Removes an existing, empty directory.
  • Removes a directory at this path, after removing all its contents. Use carefully!
  • Removes a file from the filesystem.
  • Renames a file or directory to a new name, replacing the original file if to already exists.
  • Changes the permissions found on a file or a directory.
  • Creates a new symbolic link on the filesystem.
  • Queries the file system metadata for a path.
  • Returns Ok(true) if the path points at an existing entity.
  • Creates a future that will open a file for writing and write the entire contents of contents to it.