Measure tail latency before raising request concurrency

Article · Measurement

The client completes more requests per second after its concurrency limit is raised. The graph looks better. Users report that some requests now take much longer.

Both observations can be true. A throughput improvement can come with longer queues and worse latency for the slow end of the workload.

Before increasing the limit again, I would measure what happens to those requests, including the time they spend waiting to start.

Keep the workload comparable

Use the same request mix, payload sizes, upstream behaviour, and measurement window. Record errors and timeouts rather than excluding them from the latency report.

Decide whether the workload sends requests at an independent arrival rate or waits for completion before sending more. A closed-loop client can reduce offered work when responses slow down. That can hide the queue a real traffic burst would create.

Neither workload model is always wrong. State which one matches the question being asked and what it cannot show.

The adaptive-concurrency article discusses limits when upstream capacity changes. This experiment should first establish how the fixed limits behave for one controlled workload.

Start timing before the local queue

Record admission wait separately from the actual operation. This is the server handler from the reproduction; the complete client also times the request through its body read:

async fn work(State(server): State<Server>) -> (HeaderMap, &'static str) {
    let queued = Gauge::enter(server.queued.clone());
    let started = Instant::now();
    let _permit = server.slots.acquire().await.unwrap();
    let queue_us = started.elapsed().as_micros();
    drop(queued);
    let _active = Gauge::enter(server.active.clone());
    server
        .peak
        .fetch_max(server.active.load(Ordering::SeqCst), Ordering::SeqCst);
    tokio::time::sleep(Duration::from_millis(20)).await;
    let mut headers = HeaderMap::new();
    headers.insert(
        "x-queue-us",
        HeaderValue::from_str(&queue_us.to_string()).unwrap(),
    );
    (headers, "ok")
}

The handler reports its queue wait in a response header. The client reads the fixture's tiny fixed body before recording a success. For an untrusted response, add a streaming byte cap before collecting it. Clients such as reqwest expose body reads separately from response headers. Stopping the timer when headers arrive can miss a slow body and release the concurrency permit too early.

A production implementation also needs cancellation and deadline accounting so timed-out requests remain visible in the outcome data. Include the body-reading policy in the workload description.

Report a distribution and sample count. A high percentile from a tiny sample can look more precise than the evidence supports. Keep raw or suitably aggregated observations available for review without exposing private request data.

Look for the first resource that saturates

The limit may protect a provider quota, a connection pool, CPU, or memory. Increasing it can move waiting to another layer instead of creating useful capacity.

Watch active requests, local queue wait, upstream latency, rejection rate, and timeouts together. If throughput barely changes while queue time grows, more in-flight work may mainly be adding delay.

Tokio's semaphore documentation explains the local permit mechanism. Its configured count is the limit you chose. It does not measure what the upstream can currently handle.

Pick the limit for the product

A background backfill may accept slower individual requests to maximize completion rate. An interactive search endpoint may prefer early rejection or a smaller queue to keep response time bounded.

Test several limits through the existing CI or performance workflow. Include an unchanged control and repeat when shared-runner variation matters. Publish measured figures only with the workload and environment that produced them.

The useful conclusion is not “more concurrency is bad.” It is a specific limit, for a specific workload, chosen using both completion rate and the experience of slower requests. Average speed alone cannot make that decision.

A small HTTP service at and beyond capacity

The runnable load fixture starts a loopback HTTP server with two work permits. Every accepted operation simulates twenty milliseconds of work. The client reads the tiny fixed response body and applies an eighty-millisecond request deadline. Each sample sends forty requests with a fixed maximum number in flight.

This is a closed-loop workload. When a request finishes or times out, the client can start another. It does not model an independent arrival stream or an unbounded production burst. Two rounds use opposite order so a single increasing sequence is not the entire comparison.

In the first round, concurrency one completed all forty calls at about 45.7 successful calls per second. Concurrency four completed all forty at about 93.7 per second, but successful p95 latency rose from 22.1 to 43.7 milliseconds. At concurrency twelve, only twelve calls succeeded and twenty-eight timed out. Successful throughput fell to about 41.6 per second. The second round at twelve completed eight and timed out thirty-two.

Those percentiles describe successful calls only. Timed-out calls are reported separately. They are not assigned an invented completed latency or removed from the outcome counts. Small samples also make p99 close to the slowest observed success, so these figures are a teaching reproduction rather than a stable service-level estimate.

Queue timing explains the movement. With four in flight, successful p95 queue wait was about twenty-one milliseconds. With twelve, it was about fifty-nine milliseconds. The two work permits stayed the bottleneck while more requests waited around them.

The eighty-millisecond deadline is a scheduling policy, not a hard real-time guarantee; an observed completion can land slightly beyond it when readiness and the timer race. Preserve the raw outcomes when choosing a real service limit. The recorded samples let you inspect both rounds instead of choosing the best one.