Notification System

Mobile apps, web clients, order and payment services, social features, authentication services, admin tools, and inbound webhooks all produce notification requests. The gateway authenticates the producer, authorizes its tenant and message class, applies rate limits, and routes the request to ingestion. Ingestion validates the payload schema and resolves trusted recipient context rather than accepting an arbitrary tenant ID from the request body.

An accepted intent receives a durable notification ID before the producer gets success. Scope its idempotency key to the producer or tenant and logical operation, and store a request hash so reuse with different content is rejected. This prevents a lost acknowledgement from creating a second intent; downstream send attempts still need their own repeat-safe handling.

Ingestion appends the accepted intent to a durable topic such as event.ingest. That log absorbs short bursts and lets routing workers recover after a restart. Workers attach locale, template version, and other needed context, then evaluate preferences, quiet hours, frequency caps, and audience eligibility. A finite deduplication window is useful for replay, but must cover the supported retry contract.

Routing produces explicit per-recipient channel work, separated by priority when the latency goals differ. For example, a login code can use reserved worker capacity while a campaign drains gradually under provider quotas. Quiet hours may defer a notification rather than discard it, so record deferred and suppressed outcomes separately. A queue cannot absorb sustained excess demand forever: monitor oldest useful work age and apply admission limits.

Each channel worker translates one routed intent into its provider protocol. Push workers target APNs, FCM, or Web Push endpoints; SMS workers account for gateway quotas, delivery receipts, and cost; email workers render content and process bounce or complaint evidence. Token buckets and bounded concurrency control attempted sends so retries do not consume unlimited provider capacity.

In-app notifications first enter a durable inbox and can then be announced over WebSocket or SSE. A reconnecting client reads the inbox even if the live hint was lost, and read positions drive unread counts. For external channels, provider acceptance only means the provider accepted responsibility under its contract; delivery to a device or opening by a person requires separate evidence where that channel exposes it.

Preferences express which notification types a recipient wants through each channel, along with timezone-aware quiet hours and frequency caps. Apply them again close enough to send time that a recently changed opt-out is respected. Cache entries need a version or bounded lifetime, and queued work needs a policy for changes that occur after it was originally accepted.

Classify urgency by product behavior rather than calling every transaction critical. A requested login code may need immediate delivery, while an order digest can wait until morning. Define which classes may bypass quiet hours, which channels are permitted for fallback, and what happens when preferences cannot be read. Repeatedly guessing an opt-in during a cache outage can turn a reliability incident into unwanted bulk messaging.

The template service combines a versioned tenant-and-channel template with validated substitution fields and the recipient locale. Persist the chosen version and experiment variant with the intent, so a retry produces the same approved content after a template deployment. Validate missing fields and encode output for its destination instead of blindly interpolating untrusted values into email HTML or links.

Large images and other rich assets belong in object storage and can be delivered through a CDN. Their access and lifetime should match the message: a private account document should not become public merely because a template references it. Preview and canary template changes before a campaign, and retain versioned rendering evidence so support can explain exactly what was sent.

Record the notification intent and each provider attempt with stable IDs, timestamps, status, and correlation context. Store evidence as append-only events and derive the current view, because callbacks may arrive more than once or out of order. Verify callback authenticity and map provider-specific states carefully: email bounce, SMS receipt, push acceptance, and client open are different observations.

Support can follow a correlation ID from the originating order through routing, provider attempts, and any inbox read event. Analytics projects this history into ClickHouse or a warehouse for delivery funnels and click-through rates without blocking delivery. Some channels cannot prove individual delivery or reading, and privacy features can distort open signals; label unknown outcomes instead of inventing complete tracking.

Retry temporary failures with capped exponential backoff, jitter, and the notification expiry. A two-minute login code should not be delivered after an hour in a retry queue. Permanent invalid-address or device-token errors should update the endpoint record and stop futile attempts. Malformed or repeatedly failing work goes to a dead-letter queue with a reason and controlled inspection or replay.

A provider timeout can mean that the message was accepted but its acknowledgement was lost. Reuse a supported provider idempotency key or reconcile the attempt before resending when possible. Switching immediately to another provider or channel may duplicate delivery, so define that delay-versus-duplicate trade-off per message class. Circuit breakers isolate failing provider regions or accounts while healthy channels continue serving.

Choose storage from the access pattern: transactional preference updates, per-user inbox history, append-only attempts, and analytical scans do different work. Redis can accelerate preference lookups, rate limits, and finite deduplication windows, while durable records support recovery beyond cache loss. Partition by tenant or recipient deliberately and enforce tenant access on derived views as well as source records.

Admin tools need scoped campaign and provider configuration permissions, versioned changes, previews, and a way to stop an unwanted campaign. Observe accepted, deferred, suppressed, provider-accepted, and confirmed-delivery rates separately, along with p99 latency and oldest queue age by priority. One healthy global average can hide a stuck critical lane or one tenant whose provider credentials stopped working.