SwadhyAI Notes

All System Design notes

System Design Patterns

Database Per Service

One shared database. Orders, inventory, and payments all need to update, but if three teams share that relational database, a users-table migration can break checkout at peak. Large platforms instead let each own its writes and schema.

means other services ask through APIs or events instead of joining that database. You trade easy SQL joins for independence, and you coordinate with sagas, events, and read models.

Checkout journey

1

Each service writes locally

Orders, users, payments own their stores.

2

Never share the other team's DB.

3

Multi-step money with compensations.

4

History views catch up via events.

What actually happens

Order API owns the orders store. When it needs customer fields, it calls Customer API. It does not open the customer database. Cross-service facts travel as events through an or path so publish cannot silently fail after commit.

Multi-step checkout cannot be one giant ACID transaction across databases. A sequences local commits and undoes partial work when payment fails after inventory reserved. Hot reads like order history often use a denormalized projection with rather than joining three live stores.

Pattern names stay the same at smaller scale even if you start with logical separation inside one cluster and only later split physical instances.

Order boundary

Customer boundary

Owned write pathSync APIEvents / read model

Try it: choose a cross-service data path

Database-per-service means Order API never opens Customer DB credentials. The question is whether you pay for a synchronous API read, maintain an event projection, or accidentally reintroduce a shared database.

Freshness requirement
Coupling
—
Consistency
—
Fit score
—
Suggested pattern
Read local projection from events

Boundary diagram

Order boundary

Order API · Order database (owned writes)

Customer boundary

Customer API · Customer database (owned writes)

Select a path to see how data should cross the boundary.

Order service needs a customer display name during checkout. Pick how it should fetch that data, then compare coupling and consistency trade-offs.

Sagas stay local. Multi-step checkout still commits inside each service database. Coordination uses compensating steps, not a distributed two-phase commit across stores.

Smell test. If two services must migrate schema in one release, you still have a distributed monolith regardless of repository count.

How to choose data ownership

1. Are boundaries stable? Prefer this pattern when service and data ownership are clear. Avoid it when one team is still rapidly reshaping a single domain model.

2. Do you need cross-service joins or money steps? Plan , read models, or a explicitly: distributed two-phase commit is rarely the happy path.

3. How much isolation do you need now? Start with schema ownership and permissions; move to separate clusters only when capacity, failure domains, or compliance demand it.

Simple rule of thumb

Aim for one bounded context per datastore; coordinate cross-service work with sagas and curated APIs. Treat schema ownership like code ownership and migrate in deliberate phases.

Ship the , say which reads may lag, and keep reconciliation as a product capability, not a weekend script.

Ownership levels

Table or schema owner

Shared instance, but one service owns writes, migrations, and access permissions.

Separate logical database

Clearer backup, credentials, and operational boundaries inside shared infrastructure.

Separate physical cluster

Independent capacity, failure domain, restore plan, and cost owner.

Separate engine

Use only when access patterns genuinely differ, such as search, graph, or time-series reads.

Shared database vs database per service

Database per service is primarily an ownership rule: a service controls its data model, migrations, and writes. Separate credentials and schemas on one database cluster can enforce that rule initially, although those services still share hardware capacity and outages. Separate clusters add physical isolation at additional cost. Merely creating several databases does not create autonomy if every service still edits its neighbors’ tables.

Shared database (anti-pattern at scale)

plaintext
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Orders   │ │ Users    │ │ Payments │
│ Service  │ │ Service  │ │ Service  │
└────┬─────┘ └────┬─────┘ └────┬─────┘
     │             │             │
     ▼             ▼             ▼
┌─────────────────────────────────────┐
│       SHARED RELATIONAL DATABASE      │
│  orders | users | payments tables    │
└─────────────────────────────────────┘
  • Schema change in "users" table breaks Orders + Payments
  • One slow query from Payments blocks Orders reads
  • Cannot scale databases independently
  • All services coupled to one engine, even when payment workloads need different access patterns
  • Single database = single point of failure for all services

Database per service (target shape)

plaintext
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Orders   │ │ Users    │ │ Payments │
│ Service  │ │ Service  │ │ Service  │
└────┬─────┘ └────┬─────┘ └────┬─────┘
     │             │             │
     ▼             ▼             ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│Relational│ │ Document │ │Wide-column│
│ (orders) │ │ (users)  │ │(payments) │
└──────────┘ └──────────┘ └──────────┘

Each service owns its schema and scales independently.

when workloads genuinely differ.

Why isolation helps

Independent scaling, blast-radius reduction, and engine choice per workload come from clear data ownership lines.

Independent Scaling

Read-heavy profile paths and write-heavy payment paths benefit from different engines and replica strategies: impossible to tune independently behind one tightly coupled schema.

Fault Isolation

Isolated failures contained to one domain preserve unrelated journeys: shared-instance coupling expands blast radius.

Technology Freedom

Different access patterns: search indexes, session caches, analytics warehouses: map to different engines without forcing everything through one relational box.

Team Autonomy

Teams ship migrations inside their boundary: less committee churn: provided compatibility contracts with dependents stay explicit.

Challenges: joins and transactions

A local SQL join can combine tables under one engine’s snapshot and transaction rules. Once records belong to separate services, a sequence of API reads may observe different moments in time and fail partway through. Decide whether the product needs a current composed view, a historical snapshot, or a deliberately lagging projection. The solution follows that meaning; replacing each join with an HTTP call can create a slower and less coherent result.

Challenge 1: No cross-service joins

Monolith: SQL joins across tables. Services: data split across stores: compose via APIs or projections.

javascript
// API Composition Pattern
OrderComposer.getOrderWithUser(orderId):
  order = orderService.getOrder(orderId)       // Call 1
  user  = userService.getUser(order.userId)    // Call 2
  return { ...order, userName: user.name }     // Combine
// Trade-off: 2 network calls vs 1 SQL join
// At scale: use a query layer to batch and parallelize

Challenge 2: Distributed transactions

sql
-- Monolith: All or nothing
BEGIN TRANSACTION
  INSERT INTO orders (...)
  UPDATE inventory SET qty = qty - 1
  INSERT INTO payments (...)
COMMIT  -- guaranteed

Each database commits locally: no silent cross-database transaction.

Wide two-phase commit is rarely the happy path: prefer sagas with compensations.

Challenge 3: Data consistency

Order placed → inventory decremented → payment charged. What if payment fails after inventory was decremented?

  • Need compensating transactions to undo partial work
  • Accept eventual consistency (not immediate)

Saga pattern

Model a business workflow as steps with explicit undo handlers instead of one giant Atomicity, Consistency, Isolation, Durability (ACID) transaction spanning databases. Use or .

Choreography saga (event-driven)

plaintext
OrderSvc ──(OrderCreated)──▶ InventorySvc
                               ──(InventoryReserved)──▶ PaymentSvc
                                ──(PaymentCharged)──▶ OrderSvc
                                  → Mark order CONFIRMED

Failure compensation (payment fails):
PaymentSvc ──(PaymentFailed)──▶ InventorySvc
  → Release reserved inventory
  ──(InventoryReleased)──▶ OrderSvc
    → Mark order CANCELLED

Orchestration saga (central coordinator)

  1. createOrder() → OrderSvc.create()
  2. reserveInventory() → InventorySvc.reserve()
  3. chargePayment() → PaymentSvc.charge()
  4. confirmOrder() → OrderSvc.confirm()

On failure at step 3:

compensateInventory() → cancelOrder()

  • Orchestrator tracks saga state in its own database
  • Easier to visualize long workflows than pure choreography
  • Coordinator HA becomes part of the design

API composition and read models

API composition fetches several owners’ answers at request time, which keeps data relatively current but adds latency and dependency failures. A materialized read model copies the fields needed for one query and updates them from events, making reads fast at the cost of storage and lag. For order history, preserve the product name and price accepted at purchase when historical truth matters; the current catalog name answers a different question.

Problem

"Show order history with product names and delivery status": data spans Orders, Products, Delivery.

API composer or query layer

javascript
async getOrderHistory(userId) {
  orders = await orderService.getByUser(userId)
  
  // Parallel fan-out for efficiency
  [products, deliveries] = await Promise.all([
    productService.getBatch(orders.map(o => o.productId)),
    deliveryService.getBatch(orders.map(o => o.deliveryId))
  ])
  
  return orders.map(order => ({
    ...order,
    product: products[order.productId],
    delivery: deliveries[order.deliveryId]
  }))
}

Command Query Responsibility Segregation (CQRS) alternative: materialized view

  • Events from Orders, Products, Delivery feed a read model
  • Denormalized projection simplifies hot reads
  • Trade-off: eventual consistency lag
  • Strong fit for dashboards and reporting slices

Data ownership math

Fan-out latency

If four service calls each take 60 ms in a particular request and run serially, those calls contribute 240 ms before overhead. Parallel execution may reduce elapsed time toward the slowest branch, but requires enough concurrent capacity. Individual p95 values do not sum into an exact composed p95. Measure the full query path and define what to return if one owner times out.

Projection storage

100M order-history rows × 1KB denormalized view is about 100GB raw. Add indexes and replication before promising cheap read models.

Backfill time

Backfilling 100M rows at 5k rows/sec takes about 5.6 hours before validation and retries. Throttle so production write paths stay healthy.

plaintext
projection_bytes = rows * bytes_per_row * replication_factor
100_000_000 * 1KB * 3 ~= 300GB replicated

backfill_time = rows / rows_per_sec
100_000_000 / 5_000 ~= 20_000s ~= 5.6h

lag_budget = acceptable_staleness - processing_delay
30s user tolerance - 8s processing delay = 22s remaining budget

api_composition_availability = product(required_call_availability)
0.999^4 ~= 99.6% before fallbacks

Event reliability and consistency contracts

Outbox and inbox

Write domain data and an outbox event in the same local transaction. Consumers keep an inbox dedupe record so replay does not apply the same business event twice. If a consumer updates its read model and records the processed event ID separately, it still has a crash gap. Put both writes in the same local transaction or use an atomic conditional update. Include the entity version when events can arrive out of order: deduplicating event IDs prevents repeats but does not stop an older distinct event from overwriting a newer projection.

Consistency contract

Say which reads are strongly consistent, which views may lag, what users see during lag, and how repair jobs reconcile mismatches.

Repair loop

Need replay or repair?

Replay

Reconcile

Operational checklist

Track event publish lag, consumer lag, dead-letter queue size, poison messages, replay progress, schema version adoption, read-model staleness, and business reconciliation errors.

Incremental migration

Move in phases: logical schemas and permissions, then read paths via APIs, then writes, reconciliation, and finally separate clusters when risk allows.

Step 1: Logical separation (shared database, separate schemas)

sql
-- logical schemas
schema "orders"  → owned by Order Service
schema "users"   → owned by User Service

Quick win: enforce ownership without new infrastructure. Revoke cross-schema write permissions and direct joins.

Step 2: Read path separation

  • Order Service stops JOINing to users table
  • Instead calls User Service API for user data
  • Shared database still exists but cross-schema reads eliminated

Step 3: Write path separation

  • Each service writes only to its own schema
  • Cross-service writes go through service APIs
  • Shared database becomes multiple logical databases

Step 4: Physical separation

  • Migrate schemas onto dedicated clusters or managed instances
  • Finalize backup, failover, and capacity ownership per service
  • Move only after reconciliation is green and rollback/restore ownership is clear

Anti-patterns

Shared tables with service names

Renaming schemas does not create ownership if other services still join, write, or migrate those tables directly.

Distributed monolith by API

Chatty synchronous calls for every field recreate join coupling with worse latency and failure behavior.

Polyglot persistence for novelty

Every new engine adds backup, monitoring, tuning, and hiring costs. Use different stores only for different workload needs.

No reconciliation story

Events and sagas need replay, dead-letter handling, idempotency, and repair jobs for inevitable partial failures.

What to say in a design review

Selection: at millions of users, use it when service boundaries and data ownership are stable; avoid it when one team is still rapidly reshaping a single domain model.

Tradeoffs: independent scale and deploys versus eventual consistency, fan-out latency, duplicated read models, and operational cost.

Failure modes: saga step failure, event loss, read model lag, out-of-order delivery, poison messages, and schema compatibility breaks.

Rollout: start with ownership permissions, remove direct joins, move writes behind APIs, backfill projections, then separate physical stores with reconciliation dashboards.

Production deep dive: ownership and recovery contract

Cross-service query contract

Use API composition when low fan-out and current answers matter. Use a materialized read model when read scale and latency justify defined projection lag, replay, schema version, and authorization at hydration.

Restore is a workflow, not a file copy

If Orders restores to 10:00 and Payments to 10:07, cross-service workflows disagree. Recover to documented points, replay durable events, then reconcile order, payment, and ledger IDs before reopening writes. For example, a payment captured at 10:05 may still exist while the restored order database has forgotten its order. Compare durable business IDs and provider records before taking new action. Reconciliation should repair missing state or escalate ambiguity, not blindly repeat payment calls. Restoring each database successfully is only the first step toward restoring a consistent business workflow.

Schema and tenant isolation

The owning team uses expand/backfill/contract migrations and compatible event versions. Tenant keys, row policy, pool/queue budgets, and shard placement must remain enforceable in every derived view.

Reconciliation is a product capability

Compare business identities and amounts, classify missing/duplicate/out-of-order states, repair idempotently, retain evidence, and expose an operator queue for cases no algorithm can decide safely. Run reconciliation in bounded batches so it does not overload the owners it is inspecting. Store the detected discrepancy, supporting versions, chosen action, and repair result. Re-running the same repair should be harmless. Some differences are expected projection lag; use an age threshold and known workflow state before classifying them as corruption.

plaintext
Migration states for one owned aggregate

SHARED_SOURCE
  -> COPY_SNAPSHOT(checkpoint)
  -> CATCH_UP_CDC(last_source_position)
  -> SHADOW_READ(compare version + semantic fields)
  -> NEW_WRITER(old store receives compatibility events)
  -> NEW_SOURCE_OF_TRUTH(reconciliation SLO healthy)
  -> REMOVE_OLD_ACCESS(after rollback window)

Invariant at every state:
one documented writer owns the aggregate version;
replayed commands/events are idempotent;
rollback never discards writes accepted by the new owner.
Recovery evidenceRequired checkDo not reopen writes until
Event positionsOutbox/CDC high-water marks are known per ownerNo unexplained source gap remains
Business reconciliationOrders, captures, refunds, and ledger totals agreeUncertain items are repaired or quarantined
Derived viewsProjection version/checkpoint and tenant policy passLag is inside the published consistency contract

Takeaway

Own schema and writes per bounded context; compose reads with APIs or projections; coordinate money steps with sagas. Migrate ownership in phases, ship the outbox, and treat reconciliation as part of the product, not an afterthought.

Primary sources

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Give services enforceable data ownership while supporting cross-service reads, sagas, schema evolution, coordinated recovery, reconciliation, and tenant isolation.

Explain the decision

How should a UI query data owned by five services?

Read the reasoning

Use API composition for fresh low-fan-out queries or a versioned materialized read model for scale; do not join their private tables.

Apply it to a production scenario

Orders restore to 10:00 but payments restore to 10:07 after an incident. How do you recover?

Read one approach

Restore by documented consistency points where possible, replay durable events, reconcile business IDs and ledger state, and keep uncertain workflows quarantined.

Work through the numbers

A read model projects 12 million 800-byte entities with two indexes adding 50% and three replicas. Estimate storage.

Read the worked calculation

12M × 800 × 1.5 × 3 ≈ 43.2 GB before database/page/WAL overhead.

Reason through a failure

What is the ownership invariant?

Read the recovery reasoning

Only the owning service writes its authoritative data; integration paths are versioned contracts, and replay/reconciliation can rebuild every derived view.