Don't fix a flaky test with a longer sleep

Article · Through the layers

The test starts a worker, sleeps for a short time, and checks its output. It passes locally and fails on a busy CI runner.

Increasing the sleep may make the failure less frequent. It also makes the test slower and leaves the original guess in place: the worker probably finished by now.

I would first ask which event the test actually needs to observe. That event often already has a natural signal.

Wait for readiness, not a duration

If the test needs a mock server to accept connections, let the server return its listening address after binding. If it needs a worker to reach a certain operation, send a readiness signal there.

For a retry-policy test, use a controlled transport that records calls and returns a chosen sequence of responses. The test should know when the first attempt happened before checking the next one.

Tokio supports paused time on its current-thread test runtime with the relevant features. Here is the retry-policy function exercised by the paused-clock test:

async fn fetch_with_retry(
    transport: &ScriptedTransport,
    budget: Duration,
) -> Result<(), FetchError> {
    let deadline = tokio::time::Instant::now() + budget;
    for _ in 0..2 {
        if tokio::time::Instant::now() >= deadline {
            return Err(FetchError::Deadline);
        }
        match transport.call().await? {
            Reply::Success => return Ok(()),
            Reply::RetryAfter(delay) => {
                if delay >= deadline.saturating_duration_since(tokio::time::Instant::now()) {
                    return Err(FetchError::Deadline);
                }
                tokio::time::sleep(delay).await;
            }
        }
    }
    Err(FetchError::Deadline)
}

The scripted transport, reply type, and error type are defined in the linked test harness; this function is an excerpt from it. The transport returns a rate-limit response asking for a thirty-second wait. The client has only five seconds available, so it must not start another attempt. For an allowed retry, separately check the recorded attempt times against the configured delay.

The existing API failure-testing article covers retries and bad pages together. Deterministic timing makes those scenarios easier to inspect.

A fake clock has a boundary too

Pausing Tokio time does not pause the standard library clock or a remote provider. A library using another time source may ignore the test's clock entirely.

Keep the policy test close to the code that owns the deadline. Inject a clock or transport where necessary. Then use a smaller set of real integration checks for sockets, cancellation, and actual timeout propagation.

Do not assert millisecond precision from a shared CI machine. Check meaningful bounds and recorded ordering. The exact completion time may vary while the important contract remains correct.

Preserve the assertion that found the problem

When fixing the test, keep the behaviour it was meant to protect. Removing the final assertion or retrying the test until it passes does not explain the failure.

Try the test under delayed scheduling or a deliberately stalled dependency. A readiness signal should still work because it reports a fact rather than predicting a duration. Put a bounded watchdog around the test so a broken signal becomes a clear failure instead of a job that hangs indefinitely.

There are valid timing tests. If the requirement is that a real operation times out, elapsed time belongs in the evidence. Even then, a controlled dependency and sensible tolerance are better than an arbitrary sleep before looking at state.

The goal is a test that waits for the thing it cares about. Once that relationship is explicit, the test usually becomes both faster and easier to trust.

A clock-controlled retry with an ordinary readiness signal

The complete fixture uses Tokio's clock throughout the retry policy. That detail is easy to miss. Pausing Tokio time would not control a deadline built from the standard library's clock.

Its first transport response asks for a thirty-second wait. The whole operation has a five-second budget. The policy returns Deadline after one attempt instead of scheduling work beyond that budget. A second transport asks for two seconds and then succeeds. The recorded call times differ by exactly two virtual seconds.

These checks concern the policy's chosen schedule. They do not claim that a real network request completes in an exact wall-clock interval. The transport is scripted, and the clock is paused. A real HTTP integration still needs bounded connection and response-body handling.

The same test also uses two one-shot channels around a worker. One announces that the worker reached the boundary. The other releases it. The observer waits for readiness, checks the value before release, then joins the worker and checks the final value of forty-two. No arbitrary sleep decides when the assertion runs.

A separate channel deliberately never sends. Waiting for it through a five-second watchdog returns a timeout. This control matters because a useful signal test should fail in a bounded way when the expected event never arrives. Otherwise a regression can turn the CI job into an unexplained hang.

Use signals for observable boundaries in your own code: a transaction started, a request reached the fake server, or a cleanup owner was created. Keep the watchdog longer than the expected scheduling delay, but make the signal establish readiness. A longer sleep alone cannot establish that the event happened.