Concurrency and Parallelism Basics

What actually happens

One core interleaving vs many cores at once

sequenceDiagram
    autonumber
    participant C as One CPU core
    participant K as Kernel scheduler
    participant A as Checkout A
    participant B as Checkout B

    rect rgb(30, 41, 59)
    Note over C,B: Concurrent, not parallel: one core interleaves
    A->>C: Run until DB wait
    C->>K: Block A, run B
    B->>C: Run until cache wait
    C->>K: Resume A
    A->>C: Finish
    end

flowchart LR
    API[Checkout API]
    API --> INV[Inventory shard]
    API --> PAY[Payments]
    API --> FRD[Fraud]
    INV --> JOIN[Join with deadline]
    PAY --> JOIN
    FRD --> JOIN
    JOIN --> ACK[Commit or compensate]

Fan-out is task parallelism. It only pays if inventory, payments, and fraud are independent enough to start together, and if the join has a deadline so one hung fraud call does not hold the checkout thread forever. Independent shards of a map-reduce job are data parallelism. A video pipeline that demuxes, encodes, and muxes on different stages is pipeline parallelism.

Amdahl, Gustafson, and why more cores stall

Amdahl’s law is the interview hammer: speedup is limited by the part you cannot split. If a request is 20% serial (lock, unique-id mint, final commit) and 80% parallelizable, eight workers cannot give you 8×.

Amdahl (fixed problem size):
  S(n) = 1 / (s + (1 - s) / n)

  s = 0.20 serial, n = 8 cores
  S = 1 / (0.20 + 0.80/8) = 1 / 0.30 ≈ 3.33×
  Not 8×. The last 20% dominates.

  As n → ∞, S → 1/s. Here the ceiling is 5×.

Gustson (scale the problem with the machine):
  Scaled speedup ≈ n - (n - 1)s
  Bigger jobs (more pixels, more shards) hide a fixed serial setup.

Design move:
  Shrink s: remove global locks, partition the hot row,
  mint IDs in batches, make the commit path short.

Gustafson's observation is the other side: if you grow the problem (more users, more tiles, more partitions) as you add machines, a fixed serial setup becomes a smaller fraction of wall time. That is why adding Kafka partitions helps a growing ingest job, and why a single-row inventory counter never gets that gift.

Little's Law: how many things are in flight

L = λ × W

Checkout API:
  λ = 2,000 requests/sec
  W = 200 ms = 0.2 s in the service
  L ≈ 400 requests in flight

If each in-flight request holds:
  1 inbound connection + 3 outbound (inventory, pay, fraud)
  ≈ 1,600 sockets on this service alone

Pool math:
  Need enough workers or async slots for L,
  plus headroom for bursts, minus a hard cap
  so a latency spike cannot create L = 2,000 × 2s = 4,000.

Latency and concurrency are coupled. When W doubles under load, L doubles unless you shed traffic. That is why timeouts and load shedding are concurrency controls, not just UX. Threading models (the companion note) pick how those L slots are represented: threads, goroutines, or event-loop callbacks.

Shared memory versus message passing

System design interviews often pretend services only message-pass. Inside each service you still share heaps, connection pools, and rate-limit counters. Name both layers. A Redis INCR is message passing to a shard that itself uses shared memory and single-threaded command execution—an intentional serial critical section that buys a simple mental model.

Races: memory-model bugs versus business bugs

A data race is two threads touching the same bytes without synchronization. A race condition is a timing-dependent correctness hole even when each load and store is “safe.” Checkouts that both read stock=1 and both write stock=0 can be race conditions with no C++ data race if each access is atomic. Databases call the same shape lost update or write skew; that story lives in the transactions note. Here the point is: atomics do not equal invariants.

Check-then-act (business race, even with a thread-safe map):

  if (!reserved.contains(seat)) {      // thread A and B both see empty
      reserved.add(seat);              // both add; last writer wins
      charge(card);                    // maybe both charge
  }

Atomic step (one critical section or one compare-and-set):

  if (reserved.addIfAbsent(seat)) {    // only one caller gets true
      charge(card);                    // still need idempotency if charge retries
  }

False sharing and cache-line ping-pong

Parallel counters sitting on the same 64-byte cache line invalidate each other's cores on every increment. Throughput collapses even though the counters are logically independent. Padding, striping, or per-core counters with periodic merge are the usual fixes. This is why “just shard the metric” can still be slow inside one process, and why a single global QPS AtomicLong on a 32-core box can become the new serial fraction.

Worked example: checkout latency budget

Sequential checkout: inventory 40 ms, payments 80 ms, fraud 60 ms, CPU 20 ms. Wall time 200 ms. Parallel fan-out of the three RPCs: wall time is CPU plus the slowest RPC, 20 + 80 = 100 ms, if you start them together and the join waits for all. You did not make inventory faster; you hid it under payments.

Sequential:  40 + 80 + 60 + 20 = 200 ms
Parallel:    20 + max(40, 80, 60) = 100 ms     (2×)

Amdahl view of the original 200 ms:
  serial CPU 20/200 = 10% if RPCs fully overlap
  remaining serial is the join, timeouts, and commit

Failure design:
  If fraud times out at 50 ms, do you fail closed,
  fail open, or proceed with a pending review flag?
  Parallelism made the happy path faster; it also
  made partial failure the common case.

Trade-offs to weigh

Choosing and explaining concurrency vs parallelism

Wrap-up