Message Queues

Try it: grow consumer lag, then drain the backlog

One partition behaves like a log lane. Producers append at the rate you choose; a consumer group commits offsets at its processing rate. Lag is log-end minus committed offset. When produce exceeds consume, lag grows and the oldest unprocessed event ages. When consume exceeds produce, spare capacity drains the backlog.

Naive path, then break it

What actually happens

Queue vs log

A task queue assigns each job to one active worker at a time and tracks whether processing finished. A retained log records events in order within partitions and lets independent consumer groups track positions. Queue acknowledgment usually advances or removes the delivery obligation; a log offset records consumer progress while the event remains until retention removes it. Choose according to whether the product needs one worker to perform a task, several independent reactions, or later replay.

Pick queues when each unit of work should be claimed once. Pick logs when many subscribers need the same history or you want replay and retention by policy.

Partitions, groups, and offsets

Ordering is per-partition. Scale partitions for parallelism; use stable keys when one entity's events must stay ordered end-to-end.

Partition sizing and hot keys

Partitions are your real parallel lanes. More consumers help only until every active partition is assigned; one hot key still runs serially if ordering for that key is required.

Capacity math: queues and logs

Backlog math turns "async scales" into a concrete plan: estimate incoming events, partition for parallelism, and prove consumers can drain spikes before users notice.

Ordering

Global FIFO across a whole topic is usually unnecessary and expensive. Design per-entity or per-partition order and accept interleaving elsewhere.

Delivery vs processing (no casual "exactly once")

Producer acknowledgment, consumer receipt, and completed business processing are separate events. A broker can durably accept a job while the worker has not begun. A worker can finish a side effect and then lose its acknowledgment, causing another delivery. This is why at-least-once transport requires processing that tolerates duplicates. The transport guarantee cannot decide whether two emails or two charges represent an acceptable outcome.

Most production paths are at-least-once delivery unless you invest in idempotent producers, transactional offset+sink commits, or broker dedup windows, and you still verify failure modes on your stack version. Processing safety is a separate promise.

# illustrative producer settings; names vary by client
idempotent_producer: true
transactional_writer_id: order-processor-1

# transactional consume-process-produce:
# read offsets + publish + commit offsets in one transaction boundary

Failure timeline: crash before ack + retry amplification

T0  Checkout API durable-acks OrderPaid#991 to broker; responds 200 to user
T1  Worker A sends receipt email; crashes before ack / offset commit
T2  Visibility timeout / rebalance → Worker B redelivers OrderPaid#991
T3  Without idempotency key → second email (delivery ≠ processing)
T4  Meanwhile payment vendor is slow; API + client + worker each retry ×3
T5  Offered load ≈ base × 3×3×3; vendor brownout deepens (retry amplification)
T6  Fix: one retry budget with jitter end-to-end; DLQ after max; upsert by order_id

DLQ and backpressure

A visibility timeout or processing lease allows a job to be retried if a worker disappears. If legitimate processing exceeds that timeout, another worker may receive the same job while the first still runs. Extend the lease with bounded heartbeats or choose an appropriate timeout, but retain idempotency because lease extension can fail. Backpressure limits accepted work or worker concurrency; a larger queue only postpones overload when arrivals remain above completion capacity.

Poison messages belong in a DLQ with metrics; growing main-queue depth should trigger scale-out or backpressure before memory pressure or retention loss.

for attempt in range(max_retries):
    try:
        process(msg); return ack
    except TransientError:
        backoff_sleep(attempt)
send_to_dlq(msg); alert_if_depth_high()

Batching and payloads

Amortize RTT with producer-side batching and compression; schema-based binary payloads shrink bandwidth and speed deserialization versus raw JSON at high volume.

Routing sketches

Routing decides which queue or binding receives a message before consumers run: broker-managed filtering trades flexibility for operational clarity of the graph.

Consumer lag

Lag is how far a consumer partition trails the log end: rising lag means you are slower than producers or failing commits; tie alerts to business retention windows.

Idempotent consumers

For a database side effect, write the stable message identifier into a deduplication table in the same transaction as the business update. A unique constraint makes concurrent attempts converge on one committed result. A marker written before the side effect can lose work after a crash; one written afterward outside the transaction can allow duplicates. External providers need their own idempotency key or a reconciliation procedure because they do not share the local database transaction.

At-least-once delivery means duplicates will arrive: make handling safe with a dedupe key or natural idempotency in your datastore before chasing exactly-once brokers alone.

def handle(msg):
    key = msg.business_id
    if store.already_processed(key):
        return ack
    with store.transaction():
        apply_side_effects(msg)
        store.mark_processed(key)
    return ack

Deepening: ops contracts for queues

Break the improved design

Trade-offs

Topic-specific references

• Apache Kafka: Message Delivery Semantics : at-most-once, at-least-once, and what "exactly-once" means in Kafka's transactional model.
• RabbitMQ: Reliability Guide : publisher confirms, consumer acks, and why redelivery is expected.
• AWS Builders' Library: Timeouts, retries, and backoff with jitter : retry amplification across layers.

Exercises