System Design Patterns
API Gateway
You open Instagram on the train and tap a profile. The app makes one clean request. Behind that request sit auth, abuse limits, routing to the right services, and maybe a merge of user, posts, and badges, without the phone knowing every internal hostname.
An is that front door. At large scale it handles huge request rates while aiming for single-digit-millisecond overhead: policy and routing, not a dumping ground for product logic.
One profile request
Decrypt HTTPS at the edge so the gateway can inspect routes and apply policy.
Apply per-route and per-user budgets before expensive backend work starts.
Validate a JWT locally when keys are cached, avoiding an auth-store hop.
Fan out to backends in parallel, merge results, and return one payload.
What actually happens
After that one tap, traffic usually hits DNS and a nearby point of presence first, then the . Inside a few milliseconds the gateway may terminate TLS, check , validate a , match a route, transform the request, and proxy to a backend, with logging done asynchronously so it does not add to the critical path.
Mobile on a slow network hurts when the client makes four serial calls for profile pieces. turns that into one round trip with parallel fan-out, but only if each dependency has a timeout and a fallback. Without those budgets, the gateway waits forever on many services and becomes a distributed monolith at the edge.
Different clients need different shapes. A path for mobile can return a tiny payload while web gets a richer one. Internally, may turn public REST/JSON into gRPC so backends stay efficient without exposing that protocol to every phone.
Follow a request through the gateway
Change one condition, send a request, then advance through the decisions. An invalid token should stop before quota; an empty quota should stop before routing. A slow order can time out even though its business outcome remains unknown. Controls apply to the next request, and the walkthrough keeps the settings used for the request you already sent.
Send a request, then choose Play or advance one stage at a time. Nothing plays automatically.
Playback advances every 1.4 seconds for reading, independently of the modeled request time. Pausing, selecting a stage, sending another request, or resetting cancels playback.
Model: established TLS; 9 ms total request-side gateway work; 5 ms reserved to respond; one required backend; no automatic retries or token refill. The cohort hash demonstrates stable routing, not cryptographic identity validation. A real gateway must derive that identity from verified credentials.
Try three experiments: send three valid requests with two tokens; route everyone to an unhealthy canary by setting its share to 100%; then reset and send an order whose backend work exceeds the deadline. Notice which failures spend quota and which reach the backend. Different products may choose different admission policies, but they should make those choices explicit.
Trust, admission, and routing are separate decisions
Establish identity before trusting its claims
Authentication asks who sent the request; authorization asks what that identity may do. A gateway can reject an expired token or a token missing an orders:write scope, but an order service still needs to check whether this caller owns the specific order. Do not turn a public X-User-Id header into trusted identity. Strip or replace identity headers at the trust boundary and authenticate the gateway-to-service hop, so an internal service knows where the verified context came from.
A signed JWT is useful only after validation of the allowed algorithm, signature, trusted issuer, intended audience, and time claims. Cached signing keys avoid a remote lookup for every request. Key rotation needs overlapping trusted keys and a bounded refresh path: an unknown key ID should not let an attacker trigger unlimited key fetches. Local validation also means revocation is not automatically immediate; choose the token lifetime and any online checks according to the route’s needs. These validation boundaries follow the JWT best current practices in RFC 8725.
Admission protects work that has not started yet
A cheap IP or connection limit can run before authentication to control anonymous abuse. A tenant or user quota must use a verified identity. Then a route-specific concurrency limit protects the backend: a request can be within its daily quota while the database is already full of expensive queries. Quota, arrival rate, and simultaneous work are different measurements, so one counter does not replace all three controls.
Suppose a tenant is allowed 100 requests per second across ten gateways. Giving each gateway its own 100-per-second bucket permits up to 1,000 in aggregate. A shared atomic counter can enforce a shared budget, but adds a dependency to admission. Allocating smaller local budgets reduces that dependency at the cost of temporarily stranded capacity or a carefully bounded overshoot during reallocation. If the shared store fails, an explicit emergency policy might admit public reads under a local cap while rejecting sensitive writes. The appropriate choice follows the route’s failure cost; it is not one universal fail-open switch.
Choose a service, then choose an endpoint
Route matching first maps a host, path, and method to a service or version. A rule might send GET /profiles to the profile service and POST /orders to the order service. After that decision, load balancing chooses one eligible instance inside the selected pool. These decisions have different failure modes: a missing route is a configuration problem, while a known route with no usable instance is an availability problem. Logs should identify the route, chosen cluster, and reason for rejection separately.
A sticky canary cohort can use hash(verified user ID) modulo 100 and compare the result with a rollout percentage. That gives a stable set of users while the hash and identity stay stable; it does not promise exactly 5% of requests when 5% of users are selected, because users generate different amounts of traffic. Decide whether an unavailable canary returns an error or falls back to stable, then test the effect on experiment measurements and compatibility. The lab deliberately makes rollback an explicit routing change.
How to choose gateway moves
Once you see the request path, three questions decide what belongs in the gateway versus in domain services.
1. Do clients need different shapes? Use a deliberate when web, mobile, and TV would otherwise over-fetch or under-fetch on one fat API.
2. Is aggregation worth the coupling? Compose when round trips dominate (especially mobile). Skip composition when one slow optional backend would freeze the whole page: set budgets and partial responses instead.
3. Can you retry safely? Idempotent reads may retry once. Mutations need an . Never blind-retry checkout.
Simple rule of thumb
Make the gateway the edge enforcement point for auth, limits, routing, and translation. Keep it horizontally scalable and mostly stateless. If checkout rules and pricing live in the gateway, you have built a distributed monolith at the edge: move that logic back into services.
From problem to mechanism
The choice above only works if the edge hop stays cheap at high request rates. Here is the same profile tap, slowed down into a pipeline. The numbers below are an illustrative warm-request budget; actual overhead must be measured with the chosen hardware, payloads, filters, and connection reuse.
Problem. Every client request must cross a shared edge. If that hop is slow, stateful, or stuffed with product logic, the whole product feels slow and every deploy risks every route.
Mechanism. Keep the thin: terminate TLS, enforce , validate a locally when possible, match a route, and proxy. Do logging asynchronously. Stay mostly stateless so any instance can handle any request.
Example request budget on an established TLS connection:
These terms sum to 1.2 ms of illustrative gateway work. They exclude DNS, connection establishment, backend execution, network transit, queueing, and a remote quota or authorization lookup. Asynchronous logging still consumes CPU and buffer space; bound its queue and define what to do when the log sink is unavailable. Adding independently measured p99 values would not produce an end-to-end p99: measure the complete request distribution.
Turn a measured limit into a fleet size
Assume a benchmark of this exact gateway configuration establishes a safe capacity of 35,000 requests per second per instance, and the pool must serve a peak of one million. The baseline is ceil(1,000,000 / 35,000) = 29 instances. Provisioning 40 gives 1.4 million requests per second of modeled capacity, or 40% capacity above the offered peak. These are instances, not CPU cores, and the estimate applies to the pool receiving that workload, not automatically to every region.
Now test the failure placement. Forty instances spread 14/13/13 across three zones leave only 26 after losing the largest zone; at the assumed limit that is 910,000 requests per second. Forty therefore fails a requirement to serve the full million during that zone loss. Forty-five spread evenly leave 30, or 1.05 million of capacity, before any additional maintenance margin. This is why a percentage called headroom must be tied to a concrete failure, not treated as proof of resilience.
Client shapes, composition, and protocols
A thin pipeline still has to serve phones, browsers, and TVs without wasting bandwidth or exposing internal RPC. That is where a , , and fit.
Backend-for-Frontend (BFF)
A backend-for-frontend provides an API shaped for one kind of client. A mobile profile screen might need a small avatar and the latest three posts, while a web screen needs larger images and extra controls. The BFF requests those pieces and returns one useful response shape. It should reuse domain services for permissions, prices, and workflow rules so changing a screen does not create a second implementation of business correctness.
API composition and aggregation
Suppose a profile screen needs user data, posts, and optional badges. Three serial mobile round trips add network delays; one gateway call can request all three backends in parallel. Server latency then depends roughly on the slowest required dependency plus gateway work. Assign a deadline to each branch, reserve time to assemble the response, and allow badges to be explicitly missing when their deadline expires. Bound fan-out so one external request cannot create unbounded internal work.
Without composition (N+1 client calls):
With gateway composition (1 client call):
Illustrative mobile path with 200 ms per client round trip:
This comparison assumes four serial client calls, backend work of 50/80/60/100 ms, and negligible extra backend-network time. Serial execution is 290 + 4 × 200 = 1,090 ms; parallel composition is 100 + 5 + 200 = 305 ms. A client able to issue all four calls in parallel would have a different baseline. Composition also creates four internal calls per external request, so saving mobile latency does not reduce the backends’ required throughput.
Protocol translation (REST → gRPC)
Protocol translation maps the public request contract to an internal RPC: field names and types, authentication context, deadlines, error codes, and streaming behavior all need deliberate mapping. JSON numbers may not safely represent every 64-bit identifier, and an internal error should not expose implementation details to clients. HTTP/2 and Protobuf can reduce some transport overhead, but translating protocols does not remove the need for compatible schemas and measured latency.
The 15 KB and 3 KB payloads are explicit example measurements, not a guaranteed JSON-to-Protobuf ratio. At one million messages per second they imply 15 GB/s and 3 GB/s of payload respectively, using decimal units. Measure serialization CPU separately; compression, schema choices, repeated field names, and the actual values all affect the result.
Failure budgets, circuit breakers, and partial responses
Composition only stays safe when each backend call has a timeout, retry rule, and fallback. Otherwise one sick dependency freezes the page and burns gateway workers.
A deadline is one finishing time shared by the request, while a timeout is a limit applied to a particular operation. If a gateway has 300 ms and uses 25 ms before dispatch, only 275 ms remain; reserving 15 ms to assemble and return the response leaves 260 ms for downstream work. Starting a fresh 300 ms clock at every hop spends time the caller never granted. Parallel branches share that remaining interval; serial branches spend it one after another.
Queueing consumes the same budget. At 5,000 requests per second, a mean in-flight time of 200 ms implies roughly 1,000 requests in flight under steady conditions. If the backend slows to two seconds, the unconstrained estimate becomes 10,000. At an illustrative 32 KB of buffered data each, that is 320 MB before runtime overhead. Bound active requests, pending requests, and buffered bytes per route so an optional dependency cannot consume the whole gateway.
A backend timeout does not establish whether a mutation committed. A stable idempotency key belongs to the logical operation, and the service that owns the effect must enforce deduplication or reconciliation. The gateway forwards this identity and preserves the remaining deadline. For streams, distinguish idle timeout from total duration and avoid buffering a response that the client expects incrementally. These are distinct controls in Envoy’s timeout documentation.
Per-route failure budgets
| Route | Backend calls | Timeout and retry | Fallback |
|---|---|---|---|
| User profile | user, orders, reviews, recommendations in parallel | 120 ms total; one retry only for idempotent reads | Show profile with missing optional sections. |
| Checkout | cart, inventory, payment session | No blind retry for mutations; use idempotency key | Return pending or clear error; do not duplicate side effects. |
| Search | search service, personalization service | 80 ms search, 30 ms personalization | Return generic ranking if personalization fails. |
Auth store failure
A protected route must not infer permission when it cannot validate the caller. Locally verifiable tokens can keep some requests working during an identity-provider outage, provided their signatures, issuer, audience, expiration, and required permissions can still be checked with valid cached keys. Revocation-sensitive operations may require a fresher authority. Distinguish that policy from deliberately public routes, which can remain available without authentication.
Rate-limit store failure
When the shared limiter is unavailable, a bounded local allowance can preserve selected low-risk traffic. However, a limit of 100 requests per second on each of fifty gateway instances permits up to 5,000 fleet-wide, not 100. Size emergency limits for the fleet and keep stricter behavior on abuse-sensitive routes. Monitor fallback activation so a degraded limiter does not silently become the permanent policy.
Optional backend failure
Return partial response with clear missing sections instead of failing the whole page.
Circuit breaking in the gateway
Budgets alone are not enough when a backend is already failing hard. Per-backend stop the slam:
When the circuit is OPEN:
- Return a cached response when one is available and still fresh enough.
- Return a degraded response, for example empty recommendations, instead of blocking.
- Return HTTP 503 with a Retry-After header so clients back off cleanly.
- Never forward the request to the failing service while the circuit stays open.
# Proxy outlier detection
consecutive_5xx: 5 # eject after 5 consecutive 5xx
interval: 10s # check every 10 seconds
base_ejection_time: 30s # min ejection duration
max_ejection_percent: 50 # max % ejected from poolPartial aggregation and route sizing
The same idea shows up in route-level budgets. The following table is another illustrative route policy, rather than an additional timeout to add to the earlier example. Each route needs explicit timeout, retry, and fallback so one slow backend cannot consume every worker.
| Route | Timeout | Retries | Fallback |
|---|---|---|---|
| /checkout | 800 ms | 0 | Fail closed with clear error |
| /profile | 300 ms | 1 idempotent GET | Stale cache up to 60 s |
| /recommendations | 150 ms | 0 | Empty list or popular defaults |
| /mobile/home | 500 ms total | Per-backend only | Partial JSON with missing sections |
Profile aggregation with one failed backend
Within a 300 ms total request budget, the gateway starts user, settings, and badge reads in parallel with 120 ms branch limits. User and settings return successfully; badges reach their branch timeout. Because the product defines badges as optional, the gateway returns the core profile with an explicit missing-section indicator and records the degraded result. It must not apply this fallback to required permission checks or silently translate a failed order into a successful response.
Still fails when… / trap
If checkout rules, pricing, and inventory logic all live in the gateway, every product change becomes an edge deployment. Keep orchestration in domain services; use the gateway for policy, routing, and bounded aggregation only.
Per-route rate limiting and canary routing
Edge policy is not only about failure. protects expensive and abuse-prone routes, and lets you ship a new backend version to a sticky minority first.
Per-route rate limiting
An illustrative user budget limits repeated index queries; a concurrency cap separately protects costly searches.
An IP limit slows some abuse, but shared NATs and distributed attackers require additional account and risk controls.
A browsing budget should allow normal scrolling while bounding automated refreshes; validate the threshold against real user behavior.
An upload count alone does not bound cost; also limit body size, bytes per second, and simultaneous uploads.
Allow trusted probes through the appropriate internal path and keep their handler cheap, without exposing an unbounded expensive public endpoint.
# Fixed-window sketch; this is not a sliding window.
# Key: ratelimit:{route}:{verified_user_id}:{window_start}
# Atomically: increment count and initialize expiry if new.
# Reject when count exceeds the configured window limit.
# Near a window boundary, two windows can allow a double burst.
# A token bucket or sliding-window design gives different burst behavior.Canary routing
Select roughly 5% of user hash buckets for a stable canary cohort:
if hash(user_id) % 100 < 5:
route → service-v2 (canary)
else:
route → service-v1 (stable)# Proxy route policy
route:
- destination: v1
weight: 95
- destination: v2
weight: 5This YAML is a conceptual weighted-route policy, not vendor configuration. Per-request 95/5 weights and a sticky user cohort are different policies: weighted requests may move one user between versions. Specify which behavior you need, ensure both versions understand shared data, and include enough samples before trusting a quiet canary.
Real implementations
The patterns above show up in several common gateway products. The labels below are role descriptions, not a ranking: each trades control, ops burden, and programming model differently.
Plugin-based gateway
Event-driven proxy runtime with plugins. Handles very high request rates with rate limiting, authentication, logging, transformations, access control, and analytics.
Managed API Gateway
A managed gateway transfers much of the proxy fleet operation to a provider and exposes routing, authentication, and integration features through configuration. Request, connection, payload, timeout, and regional quotas vary by product and account, so consult the selected product’s current limits before sizing it. Include request charges and data transfer in the estimate; delegating fleet management does not make backend capacity or application reliability automatic.
Programmable proxy
Layer 7 proxy and mesh sidecar. Advanced load balancing, circuit breaking, RPC-to-JSON transcoding, observability, outlier detection, retries, and dynamic xDS config without restarts.
Non-blocking gateway
Non-blocking, event-driven design for tens of billions of API requests per day. Filters for auth, routing, rate limiting, and canary rollout. Replacing a blocking gateway can improve throughput significantly.
Operate the gateway without making every change a global outage
Separate configuration distribution from request handling
The data plane is the fleet of proxies handling requests. The control plane distributes routes, certificates, limits, and policies to that fleet. Request handling should not require the control plane to answer every time. Each proxy can keep a validated configuration snapshot, apply an update atomically, and report its active version. If a proposed route is invalid, preserving the last valid snapshot can keep existing traffic working while operators fix the update.
A cached configuration is not permission to ignore expired credentials or an emergency revocation forever. Define which configuration can remain useful during disconnection and which trust decisions have a strict expiry. Canary a policy on a small proxy group, inspect route-specific errors, then expand. Keep a known-good version for rollback and detect a fleet split where half the gateways authorize or route the same request differently. An HTTP proxy’s filter order and route configuration are explicit parts of its request behavior, as shown in Envoy’s HTTP connection manager.
Deploy by draining and observe the reason for each failure
A gateway process may be mostly stateless while still holding thousands of live connections and in-flight requests. During a deployment, remove it from new admission, let bounded work finish, and use the supported protocol’s graceful shutdown behavior for long-lived connections. A WebSocket needs an application reconnect and replay plan; replacing a process does not transfer its socket to a new one. Clients should reconnect with jitter so a rolling deployment does not create a synchronized surge.
Measure gateway overhead separately from upstream time, then track rejected authentication, denied authorization, quota rejection, no healthy endpoint, upstream reset, timeout, and intentional partial responses. An aggregate 5xx graph cannot tell whether the proxy exhausted buffers or the order service failed. Attach a request ID, route, configuration version, and upstream attempt count; redact tokens and personal payloads. In a drill, slow the quota store, remove an upstream pool, and push an invalid route independently, then verify that the observed failure stays within the intended route and capacity boundary.
Choosing and explaining API gateway
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Make the gateway the edge control point for auth, rate limits, routing, and protocol translation.
Keep business logic out of the generic gateway unless it is a deliberate BFF layer.
Design it as horizontally scalable because every request crosses it.
When it belongs
Many clients need a stable API over changing backend services.
Cross-cutting concerns must be applied consistently at the edge.
Mobile, web, and partner APIs need different aggregation shapes.
How it fails
Gateway becomes a single choke point.
Slow downstream services consume gateway resources.
Bad auth or rate-limit config affects every service.
Trade-offs to weigh
Central policy control versus another hop on the request path.
BFF flexibility versus duplicated gateway code per client.
Gateway caching improves latency but risks stale personalized data.
Common misunderstandings
Putting domain workflows in the generic gateway.
Ignoring backpressure, request timeouts, and per-route limits.
Forgetting multi-region gateway failover.
Going deeper
API gateway
The mechanism and its boundary
A gateway earns its place when client diversity, shared policy, and backend churn become larger risks than one extra network hop. Keep it mostly stateless, make every dependency bounded by timeouts, and move domain orchestration to product services unless aggregation is purely request shaping.
Request path with bounded work
Client
Gateway policy
Route
Service
Work through the sizing assumptions
When it belongs
Many clients need stable contracts over a changing service graph. Auth, throttling, routing, request shaping, and observability must be consistent. The design needs canary routing, versioning, or client-specific aggregation.
Assumptions that matter
Traffic is bursty and multi-region, not a flat daily average. Most requests are independent and can be routed by stateless instances. Per-route limits exist because hot APIs fail differently.
Easy mistakes
Business workflows placed in the gateway couple unrelated products to one deployment and make the shared entry point harder to recover. Retries without a route-specific deadline and concurrency budget multiply traffic precisely when the downstream service is struggling. A personalized response cache must include the relevant identity and authorization context in its key and have an invalidation policy, or one customer can receive another customer’s data or stale permissions.
How it fails
Gateway saturation makes healthy services appear down. A bad policy push blocks every route at once. Slow downstreams occupy gateway workers until unrelated APIs degrade.
Trade-offs to name
A shared gateway makes authentication and rate limits consistent, but a bad policy can affect every route, so canary policy changes and preserve a fast rollback. The extra proxy hop costs latency and capacity while removing duplicated routing logic from clients; it is useful when many clients otherwise need to track a changing service graph. Aggregating a few backend responses can reduce client round trips, but the gateway then depends on their response shapes and failure behavior, so keep domain workflows inside the services that own them.
Wrap-up
You have seen the edge hop, the client-shape tools, and the failure controls. Use the takeaway below as the interview-ready summary, then follow the primary sources for deeper product and threat-model detail.
Key takeaway
The gateway is the critical edge hop: keep overhead tiny, scale horizontally, and use BFF and composition where clients need them. Translate protocols for efficiency, canary at the edge for safe deploys, and break circuits so one sick backend cannot take everything down. Stateless policy wins; domain workflows belong elsewhere.
Primary sources
- Envoy Proxy: What is Envoy : L7 edge proxy building block used by many programmable gateways and meshes.
- OWASP API Security Top 10 (2023) : gateway-enforced API threats that belong in edge policy, not only in services.
- Kubernetes Gateway API : modern ingress and gateway contracts for routing configuration in clusters.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Separate gateway data/control planes and define authentication context, routing, normalization, body limits, streaming, quotas, retries, and high availability.
Explain the decision
Which responsibilities should not accumulate in an API gateway?
Read the reasoning
Domain workflows and business data joins belong in services/BFFs; the gateway should enforce cross-cutting edge policy without becoming a distributed monolith.
Apply it to a production scenario
The gateway control plane is unavailable but data-plane instances still run. What happens?
Read one approach
Serve validated last-known-good configuration, reject unsafe new config, alert on staleness, and preserve an independently tested rollback path.
Work through the numbers
A gateway handles 120,000 QPS at 3 ms CPU/request. Estimate CPU cores at full utilization.
Read the worked calculation
360 CPU-seconds/s, roughly 360 cores; add utilization target, TLS, logging, and zone-failure headroom.
Reason through a failure
What must trusted identity propagation guarantee?
Read the recovery reasoning
Only an authenticated gateway can set the signed identity context, services validate it, and inbound spoofed headers are stripped.
Sources and further study
Continue with Service Discovery.