Module protobuf_native::io

source ·
Expand description

Auxiliary classes used for I/O.

The Protocol Buffer library uses the classes in this package to deal with I/O and encoding/decoding raw bytes. Most users will not need to deal with this package. However, users who want to adapt the system to work with their own I/O abstractions – e.g., to allow Protocol Buffers to be read from a different kind of input stream without the need for a temporary buffer – should take a closer look.

§Zero-copy streams

The ZeroCopyInputStream and ZeroCopyOutputStream interfaces represent abstract I/O streams to and from which protocol buffers can be read and written.

These interfaces are different from classic I/O streams in that they try to minimize the amount of data copying that needs to be done. To accomplish this, responsibility for allocating buffers is moved to the stream object, rather than being the responsibility of the caller. So, the stream can return a buffer which actually points directly into the final data structure where the bytes are to be stored, and the caller can interact directly with that buffer, eliminating an intermediate copy operation.

As an example, consider the common case in which you are reading bytes from an array that is already in memory (or perhaps an mmaped file).

With classic I/O streams, you would do something like:

let mut buffer = [0; BUFFER_SIZE];
input.read(&mut buffer);
// Do something with `buffer`.

Then the stream basically just calls memcpy to copy the data from the array into your buffer. With a ZeroCopyInputStream, you would do this instead:

let buffer = input.next();
// Do something with `buffer`.

Here, no copy is performed. The input stream returns a slice directly into the backing array, and the caller ends up reading directly from it. If you want to be able to read the old-fashioned way, you can create a CodedInputStream or CodedOutputStream wrapping these objects and use their Read/Write implementations. These will, of course, add a copy step, but the coded streams will handle buffering so at least it will be reasonably efficient.

§Coded streams

The CodedInputStream and CodedOutputStream classes, which wrap a ZeroCopyInputStream or ZeroCopyOutputStream, respectively, and allow you to read or write individual pieces of data in various formats. In particular, these implement the varint encoding for integers, a simple variable-length encoding in which smaller numbers take fewer bytes.

Typically these classes will only be used internally by the protocol buffer library in order to encode and decode protocol buffers. Clients of the library only need to know about this class if they wish to write custom message parsing or serialization procedures.

For those who are interested, varint encoding is defined as follows:

The encoding operates on unsigned integers of up to 64 bits in length. Each byte of the encoded value has the format:

  • bits 0-6: Seven bits of the number being encoded.

  • bit 7: Zero if this is the last byte in the encoding (in which case all remaining bits of the number are zero) or 1 if more bytes follow. The first byte contains the least-significant 7 bits of the number, the second byte (if present) contains the next-least-significant 7 bits, and so on. So, the binary number 1011000101011 would be encoded in two bytes as “10101011 00101100”.

In theory, varint could be used to encode integers of any length. However, for practicality we set a limit at 64 bits. The maximum encoded length of a number is thus 10 bytes.

Structs§

Traits§

  • ZeroCopyInputStream
    Abstract interface similar to an input stream but designed to minimize copying.
  • ZeroCopyOutputStream
    Abstract interface similar to an output stream but designed to minimize copying.