Retries, Deadlines, and Jitter

1. A timeout creates uncertainty about the outcome

Imagine sending “create order O-123.” The server commits the order, then the response is lost. From the client's perspective this looks like a server that never accepted the request. A timeout proves that the caller did not receive a usable response before its timer expired; it does not prove that the business operation failed.

This distinction determines retry safety. Repeating a pure lookup usually does not create another business effect, although it still consumes resources and may return newer data. Repeating an unprotected increment, charge, or job submission can produce two effects. A stable idempotency key should identify the same intended operation across attempts, bind to the same request meaning, and let the server return or reconcile the earlier outcome. A new key on every attempt describes new operations and defeats deduplication.

The server must make recording that identity and applying the effect atomic within the relevant boundary, or use a protocol that safely coordinates them. Otherwise it can crash after applying the effect but before remembering the key. Retention also matters: a retry after the deduplication record expires needs an explicit contract. AWS explains these design issues in Making retries safe with idempotent APIs.

2. A deadline limits the journey; a timeout limits one wait

An end-to-end deadline is the latest useful completion time for the logical request. A per-attempt timeout limits one attempt within that journey. Connection establishment, TLS negotiation, waiting for a connection-pool slot, receiving headers, and waiting between response chunks may have separate controls. A read-idle timeout can keep restarting as bytes arrive, so it does not necessarily bound total response duration.

For checkout with an 800 ms budget, spending 500 ms waiting for a connection leaves only 300 ms for everything afterward. Giving the network call a fresh 800 ms timeout after that wait breaks the original contract. Likewise, each downstream service must work within the remaining budget rather than restart the timer. Cancellation should interrupt local queueing and backoff as well as network I/O.

Use a monotonic timer to track elapsed duration locally, so wall-clock adjustments do not extend or shrink the wait. Across machines, naive comparison of absolute timestamps can be distorted by clock skew. A protocol can propagate a remaining timeout with elapsed time accounted for. gRPC documents this approach, along with language-specific propagation behavior, in its deadline guide. Verify what your actual client library includes in its timeout scope.

3. Allocate time along the critical path

If a request performs authentication, an inventory lookup, and an order write sequentially, their elapsed times add. If two independent lookups run concurrently, that stage waits for the slower required result, not the sum of both durations. Concurrency reduces elapsed time only when the work is genuinely independent; it can still increase total resource use.

Keep a small reserve for response serialization, cleanup, and returning an error before the upstream caller gives up. A useful downstream attempt needs enough remaining time to plausibly finish. Starting another 200 ms operation with only 15 ms left usually creates work that cannot benefit the caller. Some policies shorten the next attempt to the remaining time; others require a minimum useful allowance. Choose deliberately and measure success near the cutoff.

With the initial settings, attempt one runs from 50 to 250 ms. A 50 ms backoff precedes attempt two, which ends at 500 ms. A third full attempt would require another 100 ms wait, 200 ms execution allowance, and 50 ms response reserve: completion would need 850 ms. “Three attempts configured” therefore does not imply that three attempts should run within an 800 ms deadline.

4. Classify errors using the operation's contract

A malformed request or a permission denial will not normally improve by sending identical bytes again immediately. A transient connection interruption may improve on a later attempt. A rate-limit response asks the caller to slow down; a conflict may require reading current state and recomputing the requested change. These are different recovery actions, even when all appear as unsuccessful responses.

HTTP's Retry-After can express a delay or a date. Honor applicable server guidance without waiting beyond the caller's remaining budget. If it asks for two seconds but checkout has 150 ms left, fail or move into an explicitly asynchronous workflow. The header is defined in RFC 9110; it does not itself establish that retrying the operation is safe.

5. Backoff reduces frequency; jitter separates callers

Exponential backoff increases the delay after successive failures, with a cap to prevent unbounded waits. For retry number r, starting at zero for the first retry, an upper bound can be min(cap, base × 2^r). With a 50 ms base and a 400 ms cap, the bounds are 50, 100, 200, 400, 400 ms. These numbers are an example policy, not universally good defaults.

If every client waits exactly the same delay, an outage can synchronize their retries into repeated spikes. Full jitter chooses a random delay between zero and the current bound. Clients that failed together then return at different times. This spreads demand; it does not create more server capacity or guarantee a particular caller succeeds. AWS demonstrates the distinction between backoff alone and jittered backoff in Exponential Backoff and Jitter.

If server guidance establishes a minimum wait, do not draw a random delay below that minimum. Also avoid sleeping first and checking the deadline afterward: determine whether the proposed wait leaves useful execution time, sleep with cancellation support, and check the budget again on waking. Scheduler delays mean an intended 50 ms sleep can take longer in practice.

6. Nested retries can multiply one request into 27 attempts

Suppose the browser allows three total attempts, the gateway allows three attempts per browser attempt, and the service allows three database attempts per gateway attempt. If failures trigger every retry, one logical request can generate 3 × 3 × 3 = 27 database attempts. This is a worst-case count under the stated nesting, not a prediction that every request does so.

flowchart LR
    U[1 logical request] --> C[Client: up to 3 attempts]
    C --> G[Gateway: up to 9 service attempts]
    G --> S[Service: up to 27 database attempts]
    S --> D[Overloaded dependency]

Place retry ownership where the caller understands effect safety and the complete time budget. Other layers may have limited transparent recovery, so inventory library and proxy behavior rather than assuming only application code retries. In gRPC, the retry machinery has its own policy, throttling, and a response-header commitment boundary; that protocol boundary is distinct from a database transaction commit. See the gRPC retry guide.

A retry budget bounds extra work across requests. For an illustrative allowance of 5 retries per 100 original requests and an original rate of 10,000 requests/s, the long-run retry allowance is 500 attempts/s. A token bucket can absorb a bounded burst, but its maximum size and scope must also be chosen. If each of ten clients independently assumes it owns a global allowance of 500 retries/s, the dependency can receive 5,000 retries/s instead.

Measure original requests separately from attempts. When overload raises latency, timeouts create retries, retries add load, and load raises latency again. Per-request attempt limits alone do not break that feedback loop under a broad incident. Admission control, bounded concurrency, and an aggregate retry budget can reduce it. Google's cascading-failure chapter discusses why retry behavior must be part of overload protection.

7. Cancellation stops wasted work only when it reaches the work

A client closing its connection does not automatically stop a server's CPU loop, database query, or queued background task. Propagate cancellation through the call chain and check it at useful boundaries. For a long computation, that can mean between batches. For a queued operation, it can mean removing work that has not started. For a database driver, verify that cancellation actually reaches the running query and how the connection is reused afterward.

Cancellation is also not rollback of every completed effect. An order already committed still exists after its request deadline expires. An asynchronous job that was durably accepted may intentionally outlive the request, in which case the API should return or expose an operation identifier. Distinguish work that is no longer useful from work whose durable result still needs to be discovered.

Streaming responses require a related decision. If the caller consumed the first 500 events, retrying the whole stream can repeat them. A resumable cursor, retained event history, and deduplication can make recovery meaningful. A short per-message idle timeout and a total session policy serve different purposes; an indefinitely active stream should not accidentally inherit a short unary-request assumption.

8. Hedging trades extra work for lower tail latency

A retry typically begins after an attempt fails or times out. A hedged request begins another attempt while the first is still running, after a chosen delay. For an eligible read, the caller can take the first valid result and cancel the remaining copy. This can help when one replica has an unusual pause and another has spare capacity. The Tail at Scale explains why rare slow components matter in large fan-out requests.

Consider a hypothetical read workload where 2% of first attempts are still outstanding after the hedge delay. Launching at most one hedge for each of those requests adds up to 2% more attempts before any further retry behavior. Whether it helps depends on the delay distribution and correlation: two attempts sharing the same saturated disk may both be slow. The extra load can make that shared bottleneck worse.

Hedges need the same effect-safety, deadline, and aggregate load checks as retries. They also require an explicit result rule: two replicas may return different versions. “First response wins” is acceptable only when either response satisfies the read contract. Keep a small concurrency cap, stop losing attempts where possible, and measure the work that continues after cancellation.

9. Express the policy as one finite state machine

A production library should handle transport-specific details, but its policy can be reviewed in a small sequence. Before every attempt, verify the remaining time and concurrency allowance. After a failure, classify it, confirm effect safety, and decide whether another attempt is justified. The following pseudocode emphasizes those decisions rather than prescribing a particular client API.

deadline = monotonic_now() + total_budget
operation_id = stable_identity_for_this_business_action

for attempt in 1 .. max_attempts_including_first:
    remaining = deadline - monotonic_now() - response_reserve
    if cancelled() or remaining < minimum_useful_attempt:
        return deadline_or_cancellation_result

    result = call(operation_id,
                  timeout = min(per_attempt_timeout, remaining),
                  cancellation = parent_cancellation)
    if result.is_success: return result
    if attempt == max_attempts_including_first: return result
    if not retryable(result) or not safe_to_repeat(result): return result

    delay = jittered_backoff_respecting_server_pushback(result)
    if delay + minimum_useful_attempt > remaining_time_minus_reserve():
        return result
    sleep_with_cancellation(delay)
    if not retry_budget.try_acquire(): return result

// Also bound concurrent attempts and propagate the remaining deadline.
// An ambiguous write result can require reconciliation, not a new write.

The budget token above is acquired after backoff so it limits retries when they are about to be dispatched; a cancelled waiter need not consume a token. A circuit breaker may reject an attempt while the dependency appears unhealthy, and a bulkhead can limit concurrent use of a resource. These controls answer different questions: a deadline bounds time, a retry budget bounds extra traffic, and concurrency admission bounds simultaneous work.