Crate turmoil

Help

Expand description

Turmoil is a framework for testing distributed systems. It provides deterministic execution by running multiple concurrent hosts within a single thread. It introduces “hardship” into the system via changes in the simulated network. The network can be controlled manually or with a seeded rng.

§Hosts and Software

A turmoil simulation is comprised of one or more hosts. Hosts run software, which is represented as a Future.

Test code is also executed on a special host, called a client. This allows for an entry point into the simulated system. Client hosts have all the same capabilities as normal hosts, such as networking support.

use turmoil;

let mut sim = turmoil::Builder::new().build();

// register a host
sim.host("host", || async {

    // host software goes here

    Ok(())
});

// define test code
sim.client("test", async {

    // we can interact with other hosts from here

    Ok(())
});

// run the simulation and handle the result
_ = sim.run();

§Networking

Simulated networking types that mirror tokio::net are included in the turmoil::net module.

Turmoil is not yet oppinionated on how to structure your application code to swap in simulated types under test. More on this coming soon…

§Network Manipulation

The simulation has the following network manipulation capabilities:

partition, which introduces a network partition between hosts
repair, which repairs a network partition between hosts
hold, which holds all “in flight” messages between hosts. Messages are available for introspection using Sim’s links method.
release, which releases all “in flight” messages between hosts

§Tracing

The tracing crate is used to emit important events during the lifetime of a simulation. To enable traces, your tests must install a tracing-subscriber. The log level of turmoil can be configured using RUST_LOG=turmoil=info.

It is possible to configure your tracing subscriber to log elapsed simulation time instead of real time. See the grpc example.

Turmoil can provide a full packet level trace of the events happening in a simulation by passing RUST_LOG=turmoil=trace. This is really useful when you are unable to identify why some unexpected behaviour is happening and you need to know which packets are reaching where.

To see this in effect, you can run the axum example with the following command:

RUST_LOG=INFO,turmoil=TRACE cargo run -p axum-example

You can see the TCP packets being sent and delivered between the server and the client:

...
2023-11-29T20:23:43.276745Z TRACE node{name="server"}: turmoil: Send src=192.168.0.1:9999 dst=192.168.0.2:49152 protocol=TCP [0x48, 0x54, 0x54, 0x50, 0x2F, 0x31, 0x2E, 0x31, 0x20, 0x32, 0x30, 0x30, 0x20, 0x4F, 0x4B, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x74, 0x79, 0x70, 0x65, 0x3A, 0x20, 0x74, 0x65, 0x78, 0x74, 0x2F, 0x70, 0x6C, 0x61, 0x69, 0x6E, 0x3B, 0x20, 0x63, 0x68, 0x61, 0x72, 0x73, 0x65, 0x74, 0x3D, 0x75, 0x74, 0x66, 0x2D, 0x38, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x3A, 0x20, 0x31, 0x30, 0xD, 0xA, 0x64, 0x61, 0x74, 0x65, 0x3A, 0x20, 0x57, 0x65, 0x64, 0x2C, 0x20, 0x32, 0x39, 0x20, 0x4E, 0x6F, 0x76, 0x20, 0x32, 0x30, 0x32, 0x33, 0x20, 0x32, 0x30, 0x3A, 0x32, 0x33, 0x3A, 0x34, 0x33, 0x20, 0x47, 0x4D, 0x54, 0xD, 0xA, 0xD, 0xA, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0x66, 0x6F, 0x6F, 0x21]
2023-11-29T20:23:43.276834Z DEBUG turmoil::sim: step 43
2023-11-29T20:23:43.276907Z DEBUG turmoil::sim: step 44
2023-11-29T20:23:43.276981Z DEBUG turmoil::sim: step 45
2023-11-29T20:23:43.277039Z TRACE node{name="client"}: turmoil: Delivered src=192.168.0.1:9999 dst=192.168.0.2:49152 protocol=TCP [0x48, 0x54, 0x54, 0x50, 0x2F, 0x31, 0x2E, 0x31, 0x20, 0x32, 0x30, 0x30, 0x20, 0x4F, 0x4B, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x74, 0x79, 0x70, 0x65, 0x3A, 0x20, 0x74, 0x65, 0x78, 0x74, 0x2F, 0x70, 0x6C, 0x61, 0x69, 0x6E, 0x3B, 0x20, 0x63, 0x68, 0x61, 0x72, 0x73, 0x65, 0x74, 0x3D, 0x75, 0x74, 0x66, 0x2D, 0x38, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x3A, 0x20, 0x31, 0x30, 0xD, 0xA, 0x64, 0x61, 0x74, 0x65, 0x3A, 0x20, 0x57, 0x65, 0x64, 0x2C, 0x20, 0x32, 0x39, 0x20, 0x4E, 0x6F, 0x76, 0x20, 0x32, 0x30, 0x32, 0x33, 0x20, 0x32, 0x30, 0x3A, 0x32, 0x33, 0x3A, 0x34, 0x33, 0x20, 0x47, 0x4D, 0x54, 0xD, 0xA, 0xD, 0xA, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0x66, 0x6F, 0x6F, 0x21]
2023-11-29T20:23:43.277097Z TRACE node{name="client"}: turmoil: Recv src=192.168.0.1:9999 dst=192.168.0.2:49152 protocol=TCP [0x48, 0x54, 0x54, 0x50, 0x2F, 0x31, 0x2E, 0x31, 0x20, 0x32, 0x30, 0x30, 0x20, 0x4F, 0x4B, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x74, 0x79, 0x70, 0x65, 0x3A, 0x20, 0x74, 0x65, 0x78, 0x74, 0x2F, 0x70, 0x6C, 0x61, 0x69, 0x6E, 0x3B, 0x20, 0x63, 0x68, 0x61, 0x72, 0x73, 0x65, 0x74, 0x3D, 0x75, 0x74, 0x66, 0x2D, 0x38, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x3A, 0x20, 0x31, 0x30, 0xD, 0xA, 0x64, 0x61, 0x74, 0x65, 0x3A, 0x20, 0x57, 0x65, 0x64, 0x2C, 0x20, 0x32, 0x39, 0x20, 0x4E, 0x6F, 0x76, 0x20, 0x32, 0x30, 0x32, 0x33, 0x20, 0x32, 0x30, 0x3A, 0x32, 0x33, 0x3A, 0x34, 0x33, 0x20, 0x47, 0x4D, 0x54, 0xD, 0xA, 0xD, 0xA, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0x66, 0x6F, 0x6F, 0x21]
2023-11-29T20:23:43.277324Z  INFO client: axum_example: Got response: Response { status: 200, version: HTTP/1.1, headers: {"content-type": "text/plain; charset=utf-8", "content-length": "10", "date": "Wed, 29 Nov 2023 20:23:43 GMT"}, body: b"Hello foo!" }
...

Here the server is sending a response, before it is delivered to, and received by the client. Note that there are three steps to each packet trace in turmoil. We see Send when a packet is sent from one address to another. The packet is then Delivered to its destination, and when the destination reads the packet it is Recv’d.

§Feature flags

regex: Enables regex host resolution through ToIpAddrs

§tokio_unstable

Turmoil uses unhandled_panic to forward host panics as test failures. See unstable features to opt in.

Modules§

net: This module contains the simulated TCP/UDP networking types.

Structs§

Builder: A builder that can be used to configure the simulation.
Datagram: UDP datagram.
LinkIter: An iterator for the link, providing access to sent messages that have not yet been delivered.
LinksIter: An iterator for the network topology, providing access to all active links in the simulated network.
SentRef: Provides a reference to a message that is currently inflight on the network from one host to another.
Sim: A handle for interacting with the simulation.

Enums§

IpVersion: The kinds of networks that can be simulated in turmoil
Protocol: Supported network protocols.
Segment: This is a simplification of real TCP.

Traits§

ToIpAddr: Converts or resolves to an IpAddr.
ToIpAddrs: Converts or resolves to one or more IpAddr values.
ToSocketAddrs: A simulated version of tokio::net::ToSocketAddrs.

Functions§

elapsed: Returns how long the currently executing host has been executing for in virtual time.
established_tcp_stream_count: Return the number of established tcp streams on the current host.
established_tcp_stream_count_on: Return the number of established tcp streams on the given host.
hold: Hold messages between two hosts, or sets of hosts, until release is called.
lookup: Lookup an IP address by host name.
lookup_many: Lookup an IP address by host name. Use regex to match a number of hosts.
partition: Partition two hosts, or sets of hosts, resulting in all messages sent between them to be dropped.
partition_oneway: Partition two hosts, or sets of hosts, in one direction.
release: The opposite of hold. All held messages are immediately delivered.
repair: Repair the connection between two hosts, or sets of hosts, resulting in messages to be delivered.
repair_oneway: Repair the connection between two hosts, or sets of hosts, in one direction.
reverse_lookup: Perform a reverse DNS lookup, returning the hostname if the entry exists.
sim_elapsed: Returns how long the simulation has been executing for in virtual time.

Type Aliases§

Result: A specialized Result type for turmoil simulations.

Crate turmoilCopy item path