System Design Reliability

Circuit Breaker Pattern

What should checkout do when Pay stops answering? Cart, inventory, and recommendations are fine, but the payment service starts timing out. Without a , every request waits, thread pools fill, and a single sick dependency becomes a across the whole checkout path.

A breaker , serves a safe , and only probes again after a cooldown. At scale, that is how the rest of the system survives a leaf outage.

Checkout when payment is sick

Calls go through while a sliding window tracks failure and slow-call rates.

Stop calling payment; return immediately so thread pools stay free.

Allow a few capped probes to test whether payment recovered.

Queue a retry later or show a clear pending checkout state.

What actually happens when payment browns out

The checkout story above is not exotic; it is the default when one leaf slows down and callers keep waiting. Cart and inventory are fine, but payment starts timing out. Callers keep waiting and retrying until thread pools fill and healthy paths die.

The naive fix is "just retry three times" with a 5s and no trip wire. Counterexample: 1_000 checkout pods × 3 retries on every 503 → 3× load on an already-sick payment leaf while feed and search starve for threads. Multi-layer amplify load. Hung calls without an outer timeout hold capacity forever. Without a , one dependency consumes the shared pool and turns a leaf outage into a .

While the is , calls go to payment and a records successes, failures, and slow calls. When failure or slow-call rate crosses a threshold, the breaker trips to and callers instead of waiting.

After a wait duration, the state becomes . A few probes check whether payment recovered. Success closes the circuit; failure opens it again. Cap those probes: thousands of clients probing at once can recreate the outage. Layer this with a timeout outside the breaker, bounded retries inside it, and a bulkhead so payment cannot consume every thread. Choose a that matches risk: never guess on charges or permission checks.

How to choose breaker behavior

Once you see why retries alone make a brownout worse, three questions decide how aggressive the breaker and its fallback should be.

1. What is unsafe to invent? Payments, authz, and irreversible writes should fail closed. Profiles, recommendations, and optional facets can return cached or partial data.

2. Are retries helping or amplifying? Put the outside so one client cannot turn three attempts into a storm on a broken service.

3. Can one dependency steal the pool? Pair the breaker with a so search and feed stay healthy while payment is in .

Simple rule of thumb

outermost, then breaker, then retry, then bulkhead, then the real call. Monitor breaker state as a first-class alert: an OPEN circuit is an incident signal, not a silent setting. In simple terms: stop calling what is clearly broken, protect the rest of the system, and probe recovery carefully.

Correctness invariant

While the breaker is OPEN, callers must not invoke the dependency, only or a declared safe . Retries for a path stay inside one budget and sit inside the breaker so an open circuit stops amplification. Half-open probe concurrency is capped (per client and ideally fleet-wide). Never fail-open on charges, authz, or irreversible writes.

From problem to mechanism: the three states

The questions above tell you when to trip; here is the state machine that enforces it. Three states, one idea: call normally, stop when sick, and probe before trusting again.

Circuit Breaker States:

CLOSED ──(failure rate > threshold)──▶ OPEN

│ Normal operation │ All requests fail immediately

▲ (wait duration elapsed) ▼

└────(N successes)──── HALF-OPEN ◀─────┘

Allow limited test requests • Success → CLOSED • Failure → OPEN

While , a tracks rates. Trip to and ; after the wait, probes recovery with capped concurrency.

javascript

class CircuitBreaker {
  state = CLOSED
  failureCount = 0
  successCount = 0
  lastFailureTime = 0
  metrics = new SlidingWindowMetrics(windowSize=10s)

  async call(fn) {
    if (state === OPEN) {
      if (now() - lastFailureTime > waitDuration) {
        state = HALF_OPEN
      } else {
        throw new CircuitOpenException()  // Fail fast!
      }
    }
    try {
      result = await fn()
      onSuccess()
      return result
    } catch (error) {
      onFailure()
      throw error
    }
  }
  // ... onSuccess, onFailure
}

Timeout, breaker, retry, and bulkhead stack

A breaker, retry policy, timeout, and concurrency limiter solve different problems. An overall deadline bounds one user operation; per-attempt timeouts prevent one attempt consuming that whole budget; retries spend a limited part of what remains; a bulkhead bounds in-flight work. Decorator order changes whether the breaker records every attempt or only the final result. Choose the order deliberately, ensure an open circuit prevents further network attempts, and never retry the breaker’s own rejection blindly.

: abort if total time > 5s so hung calls cannot stall forever.
: if downstream is broken instead of waiting.
: retry transient failures with backoff, inside the breaker budget.
: limit concurrent calls so one dependency cannot steal the pool.
Actual HTTP/gRPC call

javascript

timeout(5s) {
  circuitBreaker(failRate=50%, wait=30s) {
    retry(maxAttempts=3, backoff=exponential) {
      bulkhead(maxConcurrent=20) {
        httpClient.get("http://payment-service/charge")
      }
    }
  }
}

Why this ordering matters

• Timeout outside CB: Prevents infinite waits when a call never returns.
• CB outside retry: Counts total failures including retries; stops storms.
• Retry outside bulkhead: Prevents retries from flooding the pool limit.
• Bulkhead innermost: Limits actual concurrent requests to the dependency.
Common mistake: retry without CB → 1000 clients × 3 retries = 3000 requests hammering a broken service

Bulkheads keep healthy paths alive

A breaker stops bad calls; a bulkhead stops a bad dependency from exhausting resources healthy paths need. Compare shared vs partitioned pools:

Without bulkhead

100 threads shared → Payment slow → 95 threads waiting → Only 5 left for everything → Total outage

With bulkhead

Payment: 20 threads, Search: 30, Profile: 25, Feed: 25 → Payment slow = 20 consumed, rest healthy

java

Bulkhead.of("payment",
  maxConcurrentCalls: 20,
  maxWaitDuration: Duration.ofMillis(500)
)
// If 20 calls in flight, new calls wait up to 500ms
// After 500ms → BulkheadFullException → CB counts as failure

Numbers to know

In-flight ≈ RPS × latency: 2_000 RPS × 5s timeout ≈ 10_000 blocked waits.
Cut timeout to 500ms → ~1_000 in-flight; frees threads for healthy paths.
Half-open stampede: 20_000 clients each probing once recreates the outage.
Retry layers without one budget: offered ≈ baseline × 2^layers (worst sync case).

plaintext

In-flight ≈ RPS × latency
2_000 RPS × 5s timeout → ~10_000 blocked calls on one dependency
Cut timeout to 500ms → ~1_000 in-flight; free threads for healthy paths

Half-open stampede:
  20_000 clients each allow 1 probe after waitDuration
  → 20_000 simultaneous hits the recovering service
  Cap permittedNumberOfCallsInHalfOpenState and jitter waitDuration

Retry amplification without breaker:
  layers = gateway + app + worker (each retries once)
  offered ≈ baseline × 2^layers  (worst sync): prefer one shared budget

Resilience4j configuration (production)

The same stack in a Java library: trip on failure and slow-call rates, wait in OPEN, then allow a few half-open probes.

java

CircuitBreakerConfig.custom()
  .failureRateThreshold(50)
  .slowCallRateThreshold(80)
  .slowCallDurationThreshold(Duration.ofSeconds(2))
  .slidingWindowSize(100)
  .minimumNumberOfCalls(20)
  .waitDurationInOpenState(Duration.ofSeconds(30))
  .permittedNumberOfCallsInHalfOpenState(5)
  .recordExceptions(IOException.class, TimeoutException.class)
  .ignoreExceptions(BusinessException.class)
  .build();

@CircuitBreaker(name = "payment", fallbackMethod = "paymentFallback")
@Retry(name = "payment")
@Bulkhead(name = "payment")
public PaymentResult chargePayment(Order order) {
    return paymentClient.charge(order);
}

Metrics that drive the trip

The stack above only trips when the rates look bad. Here is what a records, and which signals should page on-call (see Numbers to know above for the pileup math those rates protect against).

Sliding window metrics

Suppose a breaker requires at least twenty observed calls before evaluating a 50% failure threshold. One failure out of one request should not trip it immediately; twelve failures among twenty eligible calls would. Exclude expected business rejections such as an invalid coupon when they do not indicate dependency health. Track slow calls separately because a dependency can exhaust callers while still eventually returning HTTP 200.

• Window: last 10 seconds, 10 buckets (1s each) of recent outcomes.
• Each bucket: successes, failures, slow_calls, and not_permitted counts.
• Failure rate = failures / (successes + failures) over the window.
• Slow call rate = slow_calls / total_calls; brownouts trip before hard fails.

Metrics to monitor

circuit_breaker_state
circuit_breaker_failure_rate
circuit_breaker_calls_total
circuit_breaker_slow_calls

Alert rules

circuit_breaker_state == 1 for 5m → page on-call; OPEN is an incident signal.
circuit_breaker_failure_rate > 0.3 → warning before the cascade spreads.
not_permitted rate > 100/s → critical; callers are already failing fast.

Fallbacks, probes, and distributed breaking

Metrics tell you when to open; a decides what the user sees next, and fleet-wide probes decide whether recovery is real. Match the fallback to risk, then choose how instances share trip state.

Fail closed

Payments, permission checks, and irreversible writes should stop rather than guess.

Stale or partial response

A safe fallback makes the missing capability explicit. A profile may show approved cached public details; search can omit an optional facet; checkout must not label a charge successful merely because the payment service is unreachable. Reusing stale authorization or inventory data can violate correctness even when stale recommendation data is harmless. Give fallbacks their own expiry and latency limits so they do not become another overloaded dependency.

Queue work

Non-urgent notifications, exports, and analytics can be accepted for later processing.

Half-open probe control

If 20K clients open probes at the same second, the dependency gets another outage. Cap probes to a small concurrency, add jitter to wait durations, and require enough successes before closing.

Pileup math

See Numbers to know above: shorter timeouts plus a breaker cut in-flight waits and free threads for healthy paths.

Testing panel

Inject 5xx, slow responses, connection resets, and total outage; verify state transitions and alerts.
Assert retries use backoff and jitter; breaker counts slow calls before pools saturate.
Review each fallback for correctness: fail-open should never authorize, charge, or delete.

How fleets share trip state

A per-process breaker is usually enough; mesh adds instance-level ejection without app changes.

Per-instance breaker

A per-process breaker records the failures observed by that process for a particular dependency or route. Busy callers may reach the minimum sample count quickly while quiet callers take longer, so the fleet does not necessarily open at one instant. This keeps the mechanism available without a shared counter service, but requires jittered recovery probes and per-dependency resource limits to keep thousands of independent callers from stampeding the backend.

Shared state (in-memory store-backed)

Shared failure information can speed coordinated response, but also adds delay, a dependency on the aggregation system, and a risk that one region’s failures block healthy traffic elsewhere. Define the scope of the signal and how old it may be before trusting it. It is often better to combine local decisions with an operational override than require a remote counter lookup for every application request.

Service mesh outlier detection

A proxy can eject individual unhealthy endpoints from its local load-balancing set after observing failures. That differs from an application breaker that rejects an entire logical dependency or uses a domain-specific fallback. Other proxies may make different decisions because they see different traffic. Inspect the mesh’s actual policy distribution and ejection scope instead of assuming one proxy’s failures immediately eject an endpoint fleet-wide.

yaml

outlier_detection:
  consecutive_5xx: 5
  interval: 10s
  base_ejection_time: 30s
  max_ejection_percent: 50

Failure timeline, and what still breaks after the fix

Putting the pieces together, here is how a payment brownout plays out without a breaker, then what still goes wrong once you have one.

Timeline: retry storm without a breaker

t=0      Checkout steady at 2_000 RPS; payment p99 = 80ms; pools healthy
t=10s    Payment brownout: p99 → 4s; 40% of calls hit 5s timeout
t=11s    Clients retry ×3 (no jitter); offered load ≈ 2_000 × 3 = 6_000 RPS
t=20s    2_000 RPS × 5s timeout ≈ 10_000 in-flight waits; shared pool saturated
t=25s    Search/feed threads starved → cascading 5xx on unrelated paths
t=30s    With breaker: failure rate trips OPEN; fail-fast; in-flight collapses
         Half-open probes capped; payment gets room to recover
Outcome  Breaker trades some checkout errors for platform survival

Counterexample to "retries make it more reliable": without a budget and an open circuit, retries multiply load on the sick dependency and healthy neighbors.

What to watch

circuit_breaker_state == OPEN for 5m → page; treat it as an incident signal.
not_permitted rate, failure rate, slow-call rate, and bulkhead rejections together.
Inject 5xx / latency and assert transitions; review fail-closed fallbacks on money/authz.

Still fails when…

Thresholds too tight flap CLOSED↔OPEN on benign blips; too loose never trip.
One breaker across unrelated routes masks payment sickness behind healthy search.
Fallbacks invent charges or permissions; uncapped half-open fleets recreate outages.
Mesh without app budgets still leaves retry storms.

Trade-offs

Failing fast protects capacity but may reject requests that would have succeeded.
Configuration complexity and false opens if windows are tiny; fallbacks need product work.
Nearest alternatives: timeouts alone, load shedding, bulkheads, mesh ejection, adaptive limits.

Choosing and explaining circuit breaker

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

A circuit breaker fails fast when a dependency is unhealthy, protecting callers and the dependency.

Use closed, open, and half-open states with rolling-window metrics.

Pair it with timeouts, bounded retries, bulkheads, and fallback behavior.

When it belongs

A downstream outage could cascade through the system.

Retries are increasing load on an already failing service.

User experience can degrade gracefully.

How it fails

Breaker opens too aggressively and causes avoidable errors.

Breaker never opens because thresholds are too loose.

Fallback returns stale or unsafe data.

Trade-offs to weigh

Failing fast protects capacity but may reject requests that would have succeeded.

Half-open probes restore service gradually but need careful concurrency limits.

Per-endpoint breakers are more precise but harder to configure.

Common misunderstandings

Adding retries without timeouts and jitter.

Sharing one breaker across unrelated dependency paths.

Not instrumenting breaker state transitions.

Try it: circuit breaker states

Send calls against a flaky backend while the breaker moves among closed, open, and half-open. Watch failure counting, fast-fail while open, and probe successes required to close again.

Backend is healthy for the next call

Breaker

State: closed

Consecutive failures (closed): 0 / 3

Half-open successes: 0 / 2

Total calls: 0
Fast failures: 0
State: closed
Backend healthy: No

Circuit is closed. Failures accumulate toward the threshold; success resets the streak in the closed state.

Going deeper

Production deep dive: a breaker is a local overload controller

A circuit breaker stops requests that are unlikely to succeed, preserving caller threads and downstream recovery capacity. It must be scoped narrowly enough that one bad endpoint does not suppress unrelated operations.

What must stay true

When the breaker is open, the caller spends no downstream concurrency on normal requests; half-open probes are bounded and are the only requests allowed to test recovery.

How the pieces fit

Closed

record rolling outcomes

Open

fail fast for 30 s

Half-open

max 3 probes

Closed

probe threshold passes

Minimum sample

20 calls

Avoid opening on a tiny, noisy sample.

Failure threshold

50%

Open when failures reach half of a 100-call rolling window.

Slow-call rule

40% > 800 ms

Open before timeouts consume every caller thread.

Full state transitions

Closed records failures and slow calls. Crossing a threshold opens until cooldown. Half-open admits a small probe cohort; enough successes close and reset the window, while any configured probe failure reopens. State changes should be visible as metrics and structured events.

Half-open concurrency

Permit only a fixed number of concurrent probes, not a percentage of unrestricted traffic. Coordinate per process or accept that 100 clients × three probes means 300 recovery calls; a proxy-level breaker may provide a better aggregate recovery budget.

Slow-call detection

Count responses above a latency boundary even if they eventually succeed. A dependency returning every call in 9.9 seconds under a 10-second timeout is operationally failed because it holds concurrency and triggers upstream deadline exhaustion.

Per-endpoint policy

Separate read from write, region from region, and expensive endpoint from cheap health endpoint. A single breaker for an entire host can make a slow report API block healthy checkout requests; extremely fine keys, however, create sparse noisy windows.

Retries and bulkheads

Retry only inside the same total deadline and charge attempts to a retry budget. Place a concurrency bulkhead before the call so a closed breaker cannot consume unlimited workers; do not retry a rejection from an open breaker on the same endpoint.

Worked example: Threshold-tuning exercise

1In the latest 100 calls, 38 failed and 27 more exceeded 800 ms. Minimum sample is met.
2Failure rate is 38%, below the 50% failure threshold. Slow-call rate is 27%, below the 40% slow threshold, so remain closed.
3The next 20 calls include 15 slow calls; assume the 20 calls that age out contained none. The sliding window now has 42 slow calls (42%), so open for 30 seconds, then permit three half-open probes.

Tune from observed baseline and user deadline. Thresholds should prevent concurrency collapse without turning a brief outlier burst into a long self-inflicted outage.

Breaker interaction failures

Scenario	Consequence	How to contain it
Every client probes at once	Recovering service is hit by a thundering herd.	Bound probes, jitter cooldown, and consider proxy-level coordination.
Retry wraps breaker call	Open endpoint is retried repeatedly or alternate capacity floods.	One retry owner, total deadline, retry budget, and endpoint-aware routing.
Fallback calls same dependency	Fallback extends the failure path.	Use cached/static degraded result isolated by its own bulkhead.

What to watch in production

Graph state, transition count, rejected calls, failure/slow-call rates, half-open probe outcomes, and fallback success.
Log the window and threshold that caused each transition so operators can distinguish policy from dependency failure.
Load-test breaker + retry + bulkhead together; individual defaults can interact into request amplification.

A scenario to reason through

A low-volume admin endpoint sees two failures out of two calls and opens for five minutes. What is wrong?

One solid approach

The minimum sample and open duration are inappropriate for sparse traffic. Increase the minimum or use a longer time window with a small bounded cooldown, and scope it separately from high-volume endpoints.

Going deeper

Circuit breaker

The mechanism and its boundary

A circuit breaker stops a caller from repeatedly spending resources on a dependency that is probably failing. It protects both sides: callers fail fast, and the dependency gets room to recover.

Breaker state machine

Closed

Open on failures

Half-open probe

Recover or reject

Work through the sizing assumptions

If a dependency timeout is 2 seconds and callers issue 5,000 requests/sec, a full outage can pin about 10,000 in-flight requests before thread or connection limits bite.

A breaker that opens after 50 failures in a 10-second window reacts faster than one based only on a 5-minute error rate.

When it belongs

A downstream service is optional, slow, or failure-prone. Retries can amplify an outage. The caller has a fallback, degraded response, or queueing path.

Assumptions that matter

Failure rates are measured over rolling windows and per dependency. Bulkheads isolate pools so one dependency cannot consume all workers. Fallback quality is acceptable for the product path.

Easy mistakes

A breaker does not rescue the requests already waiting on a slow dependency, so each call also needs a timeout and a bound on concurrent work. A fallback that calls another overloaded service can move or amplify the incident instead of containing it. One shared breaker for unrelated routes lets a failing operation disable healthy traffic, so group calls by the failure boundary the breaker should isolate.

How it fails

Half-open probes are too many and trigger another outage. Breaker thresholds hide partial data corruption by returning stale fallback. All clients retry at the same time when the breaker closes.

Trade-offs to name

An open breaker releases caller resources quickly, but can reject a request that the dependency would now have served, so bounded half-open probes are needed to discover recovery. A fallback can preserve part of the user experience, but its stale or incomplete result is acceptable only when the product’s correctness rules permit it. Low failure thresholds protect capacity sooner while reacting more strongly to small noisy samples, so choose a minimum sample and thresholds for each dependency and operation.

Wrap-up

You have seen the brownout, the state machine, and the layered stack. Use these exercises to check the ideas stuck, then follow the references for the primary sources.

Exercises

Pileup sizing. Checkout is 3_000 RPS with a 3s timeout. Estimate in-flight during a total payment hang. Propose timeout + bulkhead sizes that keep 70% of the thread pool for non-payment work.
Retry amplification. Gateway retries twice; app retries twice; breaker is inside the app only. Sketch amplification and fix the stack so the invariant holds.
Fallback choice. Choose fail-closed vs cached fallback for: charge card, show recommendations, send marketing email. Name the metric that pages when OPEN.

Topic-specific references

Michael T. Nygard: Release It! (Circuit Breaker pattern).
Resilience4j documentation: CircuitBreaker, Bulkhead, Retry configuration and metrics.
Google SRE Book: Cascading Failures / Handling Overload (retry amplification and load shedding).

Key takeaway

Circuit breakers are how checkout survives a sick payment leaf. Layer them with timeouts (outermost), retries, and bulkheads (innermost). Use sliding-window metrics, safe fallbacks, and capped half-open probes. Monitor OPEN as a critical alert, and remember mesh complements per-client breakers.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Tune closed/open/half-open transitions by slow/error rate, volume, endpoint, concurrency, retry/bulkhead interaction, and measurable recovery.

Explain the decision

Why should a circuit breaker often be per endpoint or dependency operation?

Read the reasoning

One failing expensive call should not disable unrelated healthy operations or hide different latency/error baselines.

Apply it to a production scenario

Half-open probes all fire at once and overload a recovering dependency. What changes?

Read one approach

Limit probe concurrency, ramp gradually, require a minimum success window, and coordinate with retry/concurrency budgets.

Work through the numbers

A breaker opens at 50% failures over at least 40 requests. How many failures trigger it at the threshold?

Read the worked calculation

20 failures in the evaluated window; window type and slow-call policy affect actual behavior.

Reason through a failure

What must fallback preserve?

Read the recovery reasoning

It cannot claim a business action succeeded when it did not; stale/read-only/default responses need explicit semantic limits.

Sources and further study

Continue with CQRS Pattern.