System Design Communication

Async vs Sync

What must finish before the order confirms, and what can wait a few seconds? Card approve-or-decline has to be . The receipt email, loyalty points, and search-index update can finish later on an path.

Healthy systems mix both: human-facing answers stay sync where latency budgets demand it; fan-out side effects go async with retries, , and a .

Checkout decision

Caller needs approve / decline now.

Optional long work returns a job id.

Email and indexing wait safely.

Poison messages leave the main path.

Try it: sync timeout storm versus async backlog

Synchronous calls block until success, failure, or a deadline. When dependency latency exceeds the deadline, threads accumulate and retries multiply load. Asynchronous intake accepts work into a queue and returns quickly; overload shows up as depth and oldest job age instead of instant timeouts.

Arrival rate: 200/sDependency latency: 400msClient deadline: 250ms

Caller threads

In flight

Dependency

Effective attempts/s

320

Timeouts

Timed out total

Compare a synchronous chain that holds a thread until a deadline with an asynchronous handoff that enqueues and returns. Advance time to see timeout storms versus queue depth growth.

Timeout cascade. In sync mode with 200 arrivals/s, 400ms dependency latency, and a 250ms deadline, advance twenty seconds. In-flight work piles up and timeouts climb because the caller gives up before the dependency can finish.

Same load, different failure shape. Switch to async with 180 workers/s and 200 arrivals/s. The API stays fast while queue depth grows at 20/s. That is often healthier for side effects, provided the product exposes pending state and you bound retention.

Model scope: simplified retry amplification and a single dependency. Real systems add bulkheads, circuit breakers, and retry budgets.

Problem → naive → break

Concrete problem. Checkout needs card approve/decline now. Receipt email, loyalty points, and search index updates must not sit inside that same wait budget, or a slow mailer fails the payment UX.

Naive system. One synchronous chain: pay → email → loyalty → index on the request thread, with unlimited retries and no deadline.

Break. Email provider p95 jumps to 8s. Threads pile up; clients retry; effective load triples; payment auth starts timing out even though the bank is healthy. Multi-layer retries amplify the outage.

Counterexample to "just make everything async." Payment approval cannot return "maybe" without a pending product state. Async without double-charges; async without durable enqueue lies with a 200 that loses work on crash.

Correctness invariant

What must never become false

A response reports final success/failure only within its , never hangs forever holding caller threads. An is returned only after the pending job is durably recorded (and usually enqueued); loss after acceptance violates the user promise. Retries never create duplicate money/side effects without . Sync graphs fail closed with timeouts and ; async graphs bound poison with a and alert on oldest age vs retention.

What actually happens

On a sync path, every hop sits inside the user's wait budget. Latency roughly sums across services; one slow leaf blocks the whole chain unless you set a and fail fast with a .

On an async path, the API's job is durable intake: write pending state, enqueue, return with a . Workers drain at their own pace; the product shows queued → processing → succeeded / failed.

Multi-step flows add another choice: spreads reactions across services; centralizes the saga. Both need clear failure tools: timeouts for RPC, retries and a for queues.

Synchronous chain

ClientGatewayService BService C

For a serial request path, latency is roughly the sum of network transit, queueing, and processing at each dependency. If checkout waits 80 ms for inventory, 200 ms for payment, and 500 ms for email, the user waits at least 780 ms before overhead. Parallel calls can reduce elapsed time to the slowest required branch, but they still consume resources and create partial-failure cases.

Async handoff

ClientAPI202 + job id

workers drain queue / log independently

An asynchronous handoff returns after work is durably accepted, while a worker finishes it later. This changes the meaning of success: accepted does not mean completed. The product needs a pending state, a way to discover the final result, and a policy for jobs that fail permanently. The queue absorbs temporary imbalance, but sustained arrivals above processing capacity still make waiting time grow.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    autonumber
    participant U as User
    participant API as API
    participant S as Downstream service
    participant Q as Queue
    participant W as Worker

    rect rgb(30, 41, 59)
    Note over U,S: Synchronous path: caller waits
    U->>API: Submit request
    API->>S: Blocking RPC with deadline
    S-->>API: Result or error
    API-->>U: Final response
    end

    rect rgb(30, 58, 50)
    Note over U,W: Asynchronous path: caller receives acceptance
    U->>API: Submit long-running work
    API->>Q: Enqueue durably
    API-->>U: 202 Accepted plus job id
    W->>Q: Pull job later
    W-->>U: Notify, webhook, or status update
    end

Failure timeline: sync timeout → retry storm

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    autonumber
    participant U as Clients
    participant API as API
    participant S as Slow dependency

    U->>API: 2k req/s
    API->>S: RPC (deadline 250ms)
    Note over S: Latency triples
    API-->>U: timeouts
    U->>API: retries (budget ignored)
    Note over API,S: Effective attempts ≈ 6k/s; pools exhaust

T0  Steady 2k req/s × 250ms ≈ 500 in-flight
T1  Dependency latency triples; deadlines start firing
T2  Clients retry 2× without budget → ~6k attempts/sec
T3  Thread/connection pools exhaust; healthy siblings brown out
T4  Fix: shorter deadlines, jittered retry budget, circuit breaker, shed optional work

What to watch

Sync health

Track p95/p99 latency, timeout rate, retry volume, and circuit breaker opens per dependency.

Async health

Track enqueue success, queue age, consumer lag, DLQ depth, and retry exhaustion.

User promise

Expose either final success/failure now, or a clear pending state with polling, webhook, or push notification.

How to choose sync vs async

1. Does the caller need the final answer now? Auth, payment approval, and interactive reads stay with deadlines.

2. Can the product show pending? If the UI can show queued / processing / done, prefer for slow, bursty, or retryable work.

3. Are duplicates acceptable? Async only with keys, dedupe tables, or safe upserts, plus a for poison.

Simple rule of thumb

Stay sync for auth, payments, and reads where the human waits. Push everything else through durable async paths. Prefer hybrid APIs that acknowledge quickly then notify: keep validation synchronous, hand off rendering and delivery.

Capacity math: sync vs async

Sync capacity is bounded by concurrent waiting work; async capacity is bounded by durable ingest, consumer drain rate, and how much delay the product can tolerate.

User request

Needs final answer now?

Yes

Sync call with deadline

Durable enqueue

Queue depth and lag

Worker fleet

Behind SLO?

Yes

Scale, throttle, or shed optional work

Notify completion

Timeout cascade

Clients retry

API threads waiting

Slow downstream service

Timeouts

More retries (loops back)

Thread and connection pool exhaustion

Async outage recovery

API accepts work

Queue backlog grows

Retries with backoff and jitter

DLQ after bounded attempts

Workers resume after dependency recovers

Backlog drains before retention window

Decision point	Math	Example
Sync concurrency	requests/sec × average latency seconds	2k req/s × 250 ms ≈ 500 in-flight calls before retries
Async ingress	events/day ÷ 86,400 × peak factor	1M users × 8 notifications/day × 10× peak ≈ 930 jobs/sec
Backlog drain	backlog ÷ (worker rate - incoming rate)	300k queued jobs ÷ (2k/s - 1k/s) = 5 minutes
Retry amplification	effective rate = original rate × (1 + retries per failed call)	2k req/s with 2 retries during failure can become 6k attempts/sec
DLQ volume	jobs/day × final failure rate × payload bytes	8M/day × 0.1% × 1 KB ≈ 8 MB/day plus headers
Retention risk	oldest message age / retention window	9 hours old on 24-hour retention means incident response is time-boxed

How to choose

Retry, backoff, and backpressure answer

Sync retries: Use short deadlines, jittered retries, and retry budgets so callers do not amplify an outage.
Async retries: Retry transient failures with backoff, route poison messages to DLQ, and make handlers idempotent.
Backpressure: When lag grows, cap producers, scale consumers within partition limits, or shed optional work before retention expires.

Communication choice matrix

Choose sync or async by the promise you make to the user: final result now, accepted pending work, eventual notification, or best-effort telemetry.

Need final answer now?

Use sync with deadlines when the caller cannot continue without success or failure.

Duplicate work acceptable?

Use async only with idempotency keys, dedupe tables, or safe upserts.

Can product show pending?

Async is easier when the UI can show queued, processing, succeeded, or failed.

Ordering required per entity?

Use partition keys or per-entity queues before adding workers.

Maximum completion age?

Translate product promise into oldest message age and backlog drain alerts.

Downstream capacity fixed?

Add backpressure, retry budgets, and load shedding instead of hiding overload behind a queue.

Synchronous paths

Each synchronous call should receive a deadline derived from the caller’s remaining time. For example, a 500 ms request cannot give three serial dependencies 500 ms each and still meet its promise. Reserve time for returning a response and stop starting work after the budget expires. Use separate connection or worker pools for unrelated dependencies so one slow service cannot occupy every available slot.

Aspect	Notes
Semantics	Caller blocks until the downstream finishes (or fails within deadline).
HTTP	Ubiquitous and debuggable; payloads can be heavier than binary RPC.
Binary RPC	Strong contracts, multiplexing on HTTP/2: great for service-to-service.
Strengths	Immediate success/failure; straightforward tracing per request.
Risks	Head-of-line blocking; slow dependencies can cascade into overload.
Typical fits	Auth, payment authorization, reads while the user is waiting.

Asynchronous paths

Buffer in durable middleware when producers burst or consumers restart: design retries, a , and duplicate handling explicitly. A worker should acknowledge a message only after its required durable work is complete. If it crashes before acknowledgement, the broker may deliver the message again, which is why the operation needs a stable idempotency key. A dead-letter queue records messages that exhaust a bounded retry policy; an owner must inspect, repair, and replay them or mark the business operation failed.

Point-to-point queue: one consumer wins each message: natural fit for task runners and job drains.
Pub/sub log: many subscriber groups advance independent offsets over the same stream.

Strengths

Decouple availability, absorb spikes, scale consumers horizontally.

Costs

Eventual consistency, ordering rules to design, and real ops tooling.

Fire-and-forget

Emit telemetry or side-effect hints without ack path, only where loss or duplication is acceptable per product rules.

Async request–reply

bridge reply queues when results matter but callers cannot block: more plumbing than RPC. A common HTTP design returns an operation ID and a status URL after accepting a report-generation job. The client polls that URL or receives a notification when the report is ready. Keep the same operation ID across retries, authorize access to its result, and retain terminal status long enough for disconnected clients to retrieve it. Otherwise a successful background job can still look lost to the user.

Async request-reply flow

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    autonumber
    participant C as Client
    participant A as API
    participant Q as Queue
    participant W as Worker
    participant S as Result store

    C->>A: Submit request
    A->>S: Store pending job with correlation id
    A->>Q: Enqueue command
    A-->>C: 202 Accepted plus job id
    W->>Q: Process command
    W->>S: Write success or failure idempotently
    C->>S: Poll status or receive completion

Use a correlation id and store pending result before enqueueing the command.
Set timeout and expiry rules so abandoned jobs do not live forever.
Handle duplicate replies by writing success or failure idempotently for the same job id.
Expose polling, webhook, push notification, or status page based on product needs.

Choreography vs orchestration

spreads logic across reacting services; centralizes flow in a durable workflow engine: debuggability vs coupling trade-off.

Choreography

Services react to events: no central conductor owns the whole saga.

OrderCreated→Payment→Inventory→Ship

Pros: loose coupling; teams ship handlers independently.
Cons: end-to-end flow is implicit: harder to visualize and debug.

Orchestration

An orchestrator records which workflow step is running and what outcome it is waiting for. If inventory is reserved but payment is rejected, it schedules release of the reservation and records whether that compensation succeeded. Persisting the workflow state allows another process to resume after a crash. Each step remains retryable; central coordination does not turn separate service commits into one database transaction.

pay()→reserve()→ship()

Pros: explicit graph; timeouts and retries live in one place.
Cons: coordinator is a dependency: needs HA, versioning, and scaling discipline.

When to use which

Use case	Pattern	Why
User login / auth	Sync	Caller waits for token / denial
Payment authorization	Sync	Immediate approve/decline needed
Interactive API reads	Sync	User expects fresh response now
Transactional email	Async	Seconds of delay acceptable
Video transcoding	Async	Long-running job off hot path
Analytics / telemetry	Async	High volume; sampling or loss may be acceptable
Search index updates	Async	Eventual consistency OK
Realtime chat delivery	Sync-style push	WebSocket/long-lived channel keeps UX instant

How to choose

How to justify sync vs async in interviews

Use sync: When the user or caller needs the answer before proceeding, such as auth, authorization, reads, or payment approval.
Use async: When work is slow, bursty, retryable, or a side effect whose completion can be observed later.
Hybrid pattern: Accept quickly, return a job id, and publish completion through polling, push, webhook, or notification when user experience permits.

Error handling

Sync

python

try:
    resp = http_call(deadline=...)
    ...
except Timeout:
    retry_with_backoff()
except UpstreamError:
    circuit_break()

Surface failures quickly to humans or orchestrators.

Async

python

try:
    handle(msg)
except TransientError:
    raise  # nack → retry
except PermanentError:
    dlq.send(msg)

DLQ depth and poison-message drills are first-class.

Latency and isolation (illustrative)

Serial RPC stacks add latencies; enqueueing returns fast while workers drain queues: throughput numbers depend on hardware and payloads, so benchmark your services instead of copying slogans.

Sync chain

A → B → C serialized: user-visible latency is roughly the sum of each hop: budget accordingly.

Async handoff

API returns after durable enqueue; workers drain at their rate: better spike absorption when semantics allow.

Failure isolation

A downstream outage backs up the queue within retention instead of failing every synchronous caller at once, still monitor depth and apply .

Async request-reply chooser

Stay synchronous when

The user cannot proceed without the final answer.
The correctness check must happen inline before committing state.
The dependency fits inside the product latency budget with headroom.

Switch to async when

Work may take seconds or minutes.
Retries must survive process crashes and deploys.
Spikes should become queue depth instead of request timeouts.

plaintext

User clicks "export report"
- If report must download now: sync is unavoidable, but cap scope and use deadlines.
- If report can arrive later: enqueue job, return 202 + job_id, surface polling/webhook/WebSocket completion.

Interview answer:
"I would keep validation synchronous, but hand off rendering and delivery asynchronously so retries and burst absorption do not block the user thread."

Break the improved design

Hybrid sync+async fixes the checkout wait budget, then fails differently: a before durable enqueue loses work on crash; workers without double-send email; queue depth hides overload until retention drops messages; multi-layer retries (browser + API + worker) still amplify load into recovering dependencies.

spreads side effects until no one owns the end-to-end timeout. Sync paths with generous deadlines and no recreate thread pile-ups during partial outages.

Trade-offs

Sync finality vs async pending

Sync is simpler for UX when the answer fits the budget; async absorbs spikes but needs status/notify plumbing.

Choreography vs orchestration

Loose coupling vs an explicit durable graph: money-adjacent flows usually want the latter.

Queue buffering

Buffers buy isolation until retention and SLOs break: pair with , not infinite depth. For example, if arrival rate is 1,000 jobs/s and workers complete only 800 jobs/s, the queue grows by 200 jobs/s, or 720,000 jobs per hour. Recovering to 1,200 jobs/s gives only 200 jobs/s of spare capacity, so that hour of backlog takes another hour to drain while new work continues. Set admission limits from the maximum acceptable wait, not merely available storage.

Exercises

1. Split checkout into sync and async

List every side effect after Pay. Which must be synchronous? For each async step, name the durable handoff, the user-visible pending state, and the idempotency key.

2. Retry budget math

Steady 2k req/s, each failed call retries twice. During a 30% error storm, estimate attempt rate and say how you cap it (budget %, jitter, breaker) without abandoning legitimate traffic.

3. 202 before enqueue?

A teammate returns 202 after writing pending state but before the broker ack. Draw the crash timeline and decide whether you wait for enqueue, use an outbox, or accept rare lost jobs, with product language.

Going deeper

Production deep dive: communication is a product contract

Synchronous calls couple availability and latency in time; asynchronous handoffs couple schemas, queues, and eventual completion. A strong design states what the caller observes while work is pending, late, duplicated, or permanently failed.

What must stay true

Every accepted asynchronous request reaches one durable terminal state: succeeded, rejected, expired, or manually resolved, and the caller can retrieve that state by correlation ID.

How the pieces fit

POST /jobs

idempotency key

202 + job ID

deadline recorded

Queue

bounded backlog

Worker result

correlation ID

Status / callback

terminal state

Arrival burst

8,000 jobs/s

Ten-minute campaign peak.

Worker service

6,000 jobs/s

Backlog grows by 2,000 jobs/s.

Peak backlog

1.2M jobs

2,000 × 600 seconds; drain needs spare capacity.

Correlation and causation

job_id identifies the durable request, correlation_id groups the customer journey, causation_id points to the message that triggered this message, and trace context links sampled spans. Never use a transient trace ID as the only business lookup key.

Async request-reply timeout

Store accepted_at, expires_at, current state, and callback attempts. Expiry does not cancel a running side effect automatically: send an explicit cancellation command or let the worker fence itself by reading current state before commit. Late success needs a stated product rule.

Backpressure

Cap queue depth/age, producer rate, in-flight work, and per-tenant share. When the oldest job would miss its SLO, reject or degrade low-priority work with a retry-after signal instead of accepting work that cannot finish.

Contract evolution

Synchronous clients can negotiate HTTP/API versions immediately; queued commands may live for days. Consumers must support old payloads until queue retention and every DLQ/replay window expire, using additive fields and tolerant readers.

Orchestration versus choreography

Choreography is loose and resilient for a few independent reactions. Orchestration makes long workflows, deadlines, compensations, and operator visibility explicit. Neither removes idempotency; choose based on workflow ownership, not fashion.

Worked example: Deadline and drain calculation

1At burst end, backlog is 1.2M jobs. Capacity returns to 10,000 jobs/s while normal arrivals are 4,000/s.
2Net drain = 6,000 jobs/s, so backlog clears in 200 seconds.
3If the completion SLO is two minutes, the system must shed earlier, add workers, or reserve priority capacity; accepting everything already violated the promise.

Queue depth is not the user contract. Convert it to oldest-job age and predicted completion time, then enforce admission before the deadline becomes impossible.

Choreography versus orchestration failure matrix

Scenario	Consequence	How to contain it
One reaction never emits its next event	Choreography silently stalls with no single owner.	Timeout monitor or switch critical workflow to durable orchestration.
Orchestrator is unavailable	New transitions pause, though completed local work remains.	Replicate durable workflow state and resume timers idempotently.
Compensation itself fails	Business state remains partially applied.	Retry with a budget, expose manual-resolution state, and never hide the debt.
Old command exits a DLQ	New consumer rejects its schema.	Version/upcast commands through the maximum replay horizon.

What to watch in production

Graph acceptance rate, oldest-job age, predicted drain time, terminal-state distribution, callback latency, and orphaned pending jobs.
Reconcile pending jobs whose deadline passed and page on requests with no state transition.
Trace one request across API, queue, worker, database, and callback using durable identifiers.

A scenario to reason through

A report job times out for the caller at 30 seconds but successfully finishes at 45 seconds. Should the result be discarded?

One solid approach

Make the contract explicit: a polling job can still become succeeded after the client wait expires; a cancellable business command must read a fenced cancelled state before committing. Transport timeout alone cannot decide business cancellation.

Takeaway

Stay for auth, payments, and reads where the human waits; push everything else through durable paths with explicit retries and . Prefer hybrid APIs that acknowledge quickly then notify via websocket/polling when long work completes. Guard sync graphs with and bulkheads, when everything calls everything synchronously, one slow leaf browns out the forest.

Topic-specific references

Google SRE: Handling Overload : load shedding, retries, and why multi-layer retries amplify outages.
RFC 9110 §15.3.3: 202 Accepted : accepted for processing; not a promise of final success.
Circuit Breaker pattern (Azure Architecture Center) : fail fast on sick dependencies instead of exhausting sync pools.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Choose synchronous calls or asynchronous messages by response dependency, latency, coupling, backpressure, failure visibility, and workflow ownership.

Explain the decision

When is synchronous communication the better choice?

Read the reasoning

When the caller needs an immediate authoritative response and the dependency chain can meet a bounded deadline without unacceptable availability coupling.

Apply it to a production scenario

An asynchronous command times out at the UI but later succeeds. What should the API expose?

Read one approach

Return an operation ID, durable accepted state, status polling or callback, cancellation semantics, and idempotent resubmission.

Work through the numbers

A synchronous path calls three independent services at p99 80, 110, and 60 ms in parallel plus 30 ms local work. Approximate p99 budget.

Read the worked calculation

At least about 140 ms using the slowest parallel branch plus local work, while correlated tails can make real p99 worse.

Reason through a failure

What must correlation IDs and workflow state prevent?

Read the recovery reasoning

Late or duplicate replies cannot mutate the wrong request or regress a terminal workflow state.

Sources and further study

Continue with Asynchronous Programming Concepts.