Timeouts, Retries & Circuit Breakers

Try it: walk a circuit breaker through its states

Each ride request calls a maps service with a timeout. Repeated timeouts open the breaker so threads fail fast and return a cached route. Recovery uses half-open probes before closing again.

What actually happens

Timeouts: knowing when to give up

A timeout is a strict time limit on a network request. If the response doesn't arrive before the timer runs out, the client aborts. Never make a network call without a timeout.

Reliability budget for a single call

Before writing retry code, decide how much time the user can spare. Then divide that time across connection, server work, one or two retries, and a fallback. A retry that finishes after the caller has already timed out is wasted load.

Deadline propagation worked example

A deadline is one shared timer for the whole user request. Each service spends from the same clock, so downstream services stop work when there is no useful time left.

sequenceDiagram
    participant U as User
    participant E as Edge
    participant A as Service A
    participant B as Service B
    participant C as Service C

    U->>E: Request budget 900 ms
    E->>A: 800 ms left, reserve 100 ms fallback
    A->>B: 300 ms sub-deadline
    A->>C: 250 ms sub-deadline
    B-->>A: Slow response
    A-->>E: Skip retry if remaining < 150 ms
    E-->>U: Cached or partial response before deadline

Retries and idempotency

When a request fails due to a temporary network blip (a transient failure), the simplest fix is to try again. But retries are dangerous if the API is not idempotent.

Retry decision matrix

Retry only when the request is safe, still useful, and unlikely to overload the weak service. Many failures should fail fast or use a fallback.

Retry storms: when good intentions melt systems

A retry storm happens when many clients interpret slowness as failure and all try again. The original problem may be small, but retries add enough extra work to prevent recovery.

sequenceDiagram
    participant C as 100k clients
    participant API as API tier
    participant DB as Slow database

    C->>API: Original requests
    API->>DB: Query load
    DB-->>API: Slow responses
    API-->>C: Timeouts
    C->>API: Retry immediately
    C->>API: User refresh / SDK retry
    API->>DB: 2-4x more queries
    DB-->>API: Saturation worsens
    API-->>C: More timeouts

Retry amplification math

Retries multiply across layers. If the edge, service, and database client each try three times, one user request can become 27 downstream attempts during the exact moment the dependency is weakest.

Exponential backoff and jitter

If a server is struggling, immediately retrying hammers it harder. Wait before retrying, and add randomness so all clients don't retry at the exact same time. Suppose 10,000 clients all wait exactly one second after a timeout. They will produce another synchronized burst one second later. Jitter spreads those retry times across a range, reducing that alignment, while a retry budget limits the total extra work. Backoff without a total deadline can still keep obsolete work alive long after the user has given up.

// Standard Exponential Backoff with Jitter
const MAX_RETRIES = 5;
const BASE_DELAY_MS = 1000;
const MAX_DELAY_MS = 30000;

async function fetchWithBackoff(url, attempt = 0) {
    try {
        return await fetchWithTimeout(url); // Assumes strict timeout
    } catch (error) {
        if (attempt >= MAX_RETRIES) throw new Error("Max retries reached");

        // Exponential backoff: 1s, 2s, 4s, 8s...
        const backoff = BASE_DELAY_MS * Math.pow(2, attempt);
        
        // Cap the maximum wait time
        const cappedBackoff = Math.min(backoff, MAX_DELAY_MS);
        
        // Add Full Jitter: random number between 0 and cappedBackoff
        const sleepTime = Math.random() * cappedBackoff;

        console.log(`Attempt ${attempt} failed. Retrying in ${Math.round(sleepTime)}ms`);
        await new Promise(res => setTimeout(res, sleepTime));
        
        return fetchWithBackoff(url, attempt + 1);
    }
}

Cascading failures

A cascading failure happens when a problem in one small part of the system spreads to healthy parts, eventually bringing the entire architecture down.

Bulkheads and capacity isolation

Bulkheads keep failure contained. Give risky dependencies separate thread pools, connection pools, queues, and rate limits so they cannot steal capacity from the critical path.

Circuit breakers: stopping the cascade

Inspired by electrical breakers, this pattern monitors external calls. If a service is failing repeatedly, the breaker trips and immediately blocks future calls, giving the dependency time to recover without overwhelming it. A breaker observes recent failures and changes behavior for calls to one dependency. It helps only when rejected calls release resources or use a cheaper fallback. An application that immediately retries elsewhere at the same total volume may simply move overload around. Keep breaker scope, minimum sample size, failure classification, and recovery probes explicit so a small noisy sample does not disable a healthy service.

stateDiagram-v2
    [*] --> CLOSED
    
    CLOSED --> OPEN : Failure rate > Threshold (e.g. 50%)
    
    OPEN --> HALF_OPEN : Timeout expires (e.g. 30s)
    OPEN --> OPEN : Fast fail incoming requests
    
    HALF_OPEN --> CLOSED : Success (Service recovered!)
    HALF_OPEN --> OPEN : Failure (Still broken)

class CircuitBreaker {
    constructor(failureThreshold, resetTimeout) {
        this.state = 'CLOSED';
        this.failures = 0;
        this.threshold = failureThreshold;
        this.timeout = resetTimeout;
        this.nextAttempt = null;
    }

    async call(apiFunction) {
        if (this.state === 'OPEN') {
            if (Date.now() > this.nextAttempt) {
                this.state = 'HALF_OPEN';
            } else {
                throw new Error("Circuit is OPEN. Fast failing!");
            }
        }

        try {
            const result = await apiFunction();
            this.reset(); // Success!
            return result;
        } catch (error) {
            this.recordFailure();
            throw error;
        }
    }

    recordFailure() {
        this.failures += 1;
        if (this.failures >= this.threshold) {
            this.state = 'OPEN';
            this.nextAttempt = Date.now() + this.timeout;
        }
    }

    reset() {
        this.failures = 0;
        this.state = 'CLOSED';
    }
}

What must never become false

Failure timeline: nested retries

t=0      Payment leaf brownout; baseline 1M user req/min
t=10s    Edge retries ×2, service ×3, DB client ×3 → up to ~18M attempts/min
t=20s    Thread pools fill; healthy neighbors time out (cascading failure)
t=30s    Circuit opens; fail-fast + cached/degraded path; amplification stops
t=40s    Half-open probes capped; dependency recovers without stampede
Outcome  Retries without budgets turn a leaf outage into a platform outage

Exercises

References

• Michael T. Nygard: Release It! (timeouts, bulkheads, circuit breakers).
• Google SRE Book: Cascading Failures / Handling Overload (retry amplification).
• AWS Architecture Blog / Builders Library: Timeouts, retries, and backoff with jitter.

Going deeper: retry ownership and adaptive concurrency