System Design Patterns
Database Per Service
One shared database. Orders, inventory, and payments all need to update, but if three teams share that relational database, a users-table migration can break checkout at peak. Large platforms instead let each own its writes and schema.
means other services ask through APIs or events instead of joining that database. You trade easy SQL joins for independence, and you coordinate with sagas, events, and read models.
Checkout journey
Each service writes locally
Orders, users, payments own their stores.
Never share the other team's DB.
Multi-step money with compensations.
History views catch up via events.
What actually happens
Order API owns the orders store. When it needs customer fields, it calls Customer API. It does not open the customer database. Cross-service facts travel as events through an or path so publish cannot silently fail after commit.
Multi-step checkout cannot be one giant ACID transaction across databases. A sequences local commits and undoes partial work when payment fails after inventory reserved. Hot reads like order history often use a denormalized projection with rather than joining three live stores.
Pattern names stay the same at smaller scale even if you start with logical separation inside one cluster and only later split physical instances.
Order boundary
Order API
Order database
Customer boundary
Customer API
Customer database
Order API needs customer data
Customer API
sync ask, never share DB
OrderPlaced / CustomerUpdated events
Order history read model
Try it: choose a cross-service data path
Database-per-service means Order API never opens Customer DB credentials. The question is whether you pay for a synchronous API read, maintain an event projection, or accidentally reintroduce a shared database.
- Coupling
- —
- Consistency
- —
- Fit score
- —
- Suggested pattern
- Read local projection from events
Boundary diagram
Order boundary
Order API · Order database (owned writes)
Customer boundary
Customer API · Customer database (owned writes)
Select a path to see how data should cross the boundary.
Order service needs a customer display name during checkout. Pick how it should fetch that data, then compare coupling and consistency trade-offs.
Sagas stay local. Multi-step checkout still commits inside each service database. Coordination uses compensating steps, not a distributed two-phase commit across stores.
Smell test. If two services must migrate schema in one release, you still have a distributed monolith regardless of repository count.
How to choose data ownership
1. Are boundaries stable? Prefer this pattern when service and data ownership are clear. Avoid it when one team is still rapidly reshaping a single domain model.
2. Do you need cross-service joins or money steps? Plan , read models, or a explicitly: distributed two-phase commit is rarely the happy path.
3. How much isolation do you need now? Start with schema ownership and permissions; move to separate clusters only when capacity, failure domains, or compliance demand it.
Simple rule of thumb
Aim for one bounded context per datastore; coordinate cross-service work with sagas and curated APIs. Treat schema ownership like code ownership and migrate in deliberate phases.
Ship the , say which reads may lag, and keep reconciliation as a product capability, not a weekend script.
Ownership levels
Table or schema owner
Shared instance, but one service owns writes, migrations, and access permissions.
Separate logical database
Clearer backup, credentials, and operational boundaries inside shared infrastructure.
Separate physical cluster
Independent capacity, failure domain, restore plan, and cost owner.
Separate engine
Use only when access patterns genuinely differ, such as search, graph, or time-series reads.
Shared database vs database per service
Database per service is primarily an ownership rule: a service controls its data model, migrations, and writes. Separate credentials and schemas on one database cluster can enforce that rule initially, although those services still share hardware capacity and outages. Separate clusters add physical isolation at additional cost. Merely creating several databases does not create autonomy if every service still edits its neighbors’ tables.
Shared database (anti-pattern at scale)
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Orders │ │ Users │ │ Payments │
│ Service │ │ Service │ │ Service │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────┐
│ SHARED RELATIONAL DATABASE │
│ orders | users | payments tables │
└─────────────────────────────────────┘- Schema change in "users" table breaks Orders + Payments
- One slow query from Payments blocks Orders reads
- Cannot scale databases independently
- All services coupled to one engine, even when payment workloads need different access patterns
- Single database = single point of failure for all services
Database per service (target shape)
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Orders │ │ Users │ │ Payments │
│ Service │ │ Service │ │ Service │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│Relational│ │ Document │ │Wide-column│
│ (orders) │ │ (users) │ │(payments) │
└──────────┘ └──────────┘ └──────────┘Each service owns its schema and scales independently.
when workloads genuinely differ.
Why isolation helps
Independent scaling, blast-radius reduction, and engine choice per workload come from clear data ownership lines.
Independent Scaling
Read-heavy profile paths and write-heavy payment paths benefit from different engines and replica strategies: impossible to tune independently behind one tightly coupled schema.
Fault Isolation
Isolated failures contained to one domain preserve unrelated journeys: shared-instance coupling expands blast radius.
Technology Freedom
Different access patterns: search indexes, session caches, analytics warehouses: map to different engines without forcing everything through one relational box.
Team Autonomy
Teams ship migrations inside their boundary: less committee churn: provided compatibility contracts with dependents stay explicit.
Challenges: joins and transactions
A local SQL join can combine tables under one engine’s snapshot and transaction rules. Once records belong to separate services, a sequence of API reads may observe different moments in time and fail partway through. Decide whether the product needs a current composed view, a historical snapshot, or a deliberately lagging projection. The solution follows that meaning; replacing each join with an HTTP call can create a slower and less coherent result.
Challenge 1: No cross-service joins
Monolith: SQL joins across tables. Services: data split across stores: compose via APIs or projections.
// API Composition Pattern
OrderComposer.getOrderWithUser(orderId):
order = orderService.getOrder(orderId) // Call 1
user = userService.getUser(order.userId) // Call 2
return { ...order, userName: user.name } // Combine
// Trade-off: 2 network calls vs 1 SQL join
// At scale: use a query layer to batch and parallelizeChallenge 2: Distributed transactions
-- Monolith: All or nothing
BEGIN TRANSACTION
INSERT INTO orders (...)
UPDATE inventory SET qty = qty - 1
INSERT INTO payments (...)
COMMIT -- guaranteedEach database commits locally: no silent cross-database transaction.
Wide two-phase commit is rarely the happy path: prefer sagas with compensations.
Challenge 3: Data consistency
Order placed → inventory decremented → payment charged. What if payment fails after inventory was decremented?
- Need compensating transactions to undo partial work
- Accept eventual consistency (not immediate)
Saga pattern
Model a business workflow as steps with explicit undo handlers instead of one giant Atomicity, Consistency, Isolation, Durability (ACID) transaction spanning databases. Use or .
Choreography saga (event-driven)
OrderSvc ──(OrderCreated)──▶ InventorySvc
──(InventoryReserved)──▶ PaymentSvc
──(PaymentCharged)──▶ OrderSvc
→ Mark order CONFIRMED
Failure compensation (payment fails):
PaymentSvc ──(PaymentFailed)──▶ InventorySvc
→ Release reserved inventory
──(InventoryReleased)──▶ OrderSvc
→ Mark order CANCELLEDOrchestration saga (central coordinator)
- createOrder() → OrderSvc.create()
- reserveInventory() → InventorySvc.reserve()
- chargePayment() → PaymentSvc.charge()
- confirmOrder() → OrderSvc.confirm()
On failure at step 3:
compensateInventory() → cancelOrder()
- Orchestrator tracks saga state in its own database
- Easier to visualize long workflows than pure choreography
- Coordinator HA becomes part of the design
API composition and read models
API composition fetches several owners’ answers at request time, which keeps data relatively current but adds latency and dependency failures. A materialized read model copies the fields needed for one query and updates them from events, making reads fast at the cost of storage and lag. For order history, preserve the product name and price accepted at purchase when historical truth matters; the current catalog name answers a different question.
Problem
"Show order history with product names and delivery status": data spans Orders, Products, Delivery.
API composer or query layer
async getOrderHistory(userId) {
orders = await orderService.getByUser(userId)
// Parallel fan-out for efficiency
[products, deliveries] = await Promise.all([
productService.getBatch(orders.map(o => o.productId)),
deliveryService.getBatch(orders.map(o => o.deliveryId))
])
return orders.map(order => ({
...order,
product: products[order.productId],
delivery: deliveries[order.deliveryId]
}))
}Command Query Responsibility Segregation (CQRS) alternative: materialized view
- Events from Orders, Products, Delivery feed a read model
- Denormalized projection simplifies hot reads
- Trade-off: eventual consistency lag
- Strong fit for dashboards and reporting slices
Data ownership math
Fan-out latency
If four service calls each take 60 ms in a particular request and run serially, those calls contribute 240 ms before overhead. Parallel execution may reduce elapsed time toward the slowest branch, but requires enough concurrent capacity. Individual p95 values do not sum into an exact composed p95. Measure the full query path and define what to return if one owner times out.
Projection storage
100M order-history rows × 1KB denormalized view is about 100GB raw. Add indexes and replication before promising cheap read models.
Backfill time
Backfilling 100M rows at 5k rows/sec takes about 5.6 hours before validation and retries. Throttle so production write paths stay healthy.
projection_bytes = rows * bytes_per_row * replication_factor
100_000_000 * 1KB * 3 ~= 300GB replicated
backfill_time = rows / rows_per_sec
100_000_000 / 5_000 ~= 20_000s ~= 5.6h
lag_budget = acceptable_staleness - processing_delay
30s user tolerance - 8s processing delay = 22s remaining budget
api_composition_availability = product(required_call_availability)
0.999^4 ~= 99.6% before fallbacksEvent reliability and consistency contracts
Outbox and inbox
Write domain data and an outbox event in the same local transaction. Consumers keep an inbox dedupe record so replay does not apply the same business event twice. If a consumer updates its read model and records the processed event ID separately, it still has a crash gap. Put both writes in the same local transaction or use an atomic conditional update. Include the entity version when events can arrive out of order: deduplicating event IDs prevents repeats but does not stop an older distinct event from overwriting a newer projection.
Consistency contract
Say which reads are strongly consistent, which views may lag, what users see during lag, and how repair jobs reconcile mismatches.
Service write model
Local transaction
Owned database
Outbox table
Publisher
Event bus
Consumer inbox dedupe
Read model
Repair loop
Need replay or repair?
Replay
Bus replays into inbox dedupe
Reconcile
Reconciliation job
Repair command back to write model
Operational checklist
Track event publish lag, consumer lag, dead-letter queue size, poison messages, replay progress, schema version adoption, read-model staleness, and business reconciliation errors.
Incremental migration
Move in phases: logical schemas and permissions, then read paths via APIs, then writes, reconciliation, and finally separate clusters when risk allows.
Step 1: Logical separation (shared database, separate schemas)
-- logical schemas
schema "orders" → owned by Order Service
schema "users" → owned by User ServiceQuick win: enforce ownership without new infrastructure. Revoke cross-schema write permissions and direct joins.
Step 2: Read path separation
- Order Service stops JOINing to users table
- Instead calls User Service API for user data
- Shared database still exists but cross-schema reads eliminated
Step 3: Write path separation
- Each service writes only to its own schema
- Cross-service writes go through service APIs
- Shared database becomes multiple logical databases
Step 4: Physical separation
- Migrate schemas onto dedicated clusters or managed instances
- Finalize backup, failover, and capacity ownership per service
- Move only after reconciliation is green and rollback/restore ownership is clear
Anti-patterns
Shared tables with service names
Renaming schemas does not create ownership if other services still join, write, or migrate those tables directly.
Distributed monolith by API
Chatty synchronous calls for every field recreate join coupling with worse latency and failure behavior.
Polyglot persistence for novelty
Every new engine adds backup, monitoring, tuning, and hiring costs. Use different stores only for different workload needs.
No reconciliation story
Events and sagas need replay, dead-letter handling, idempotency, and repair jobs for inevitable partial failures.
What to say in a design review
Selection: at millions of users, use it when service boundaries and data ownership are stable; avoid it when one team is still rapidly reshaping a single domain model.
Tradeoffs: independent scale and deploys versus eventual consistency, fan-out latency, duplicated read models, and operational cost.
Failure modes: saga step failure, event loss, read model lag, out-of-order delivery, poison messages, and schema compatibility breaks.
Rollout: start with ownership permissions, remove direct joins, move writes behind APIs, backfill projections, then separate physical stores with reconciliation dashboards.
Production deep dive: ownership and recovery contract
Cross-service query contract
Use API composition when low fan-out and current answers matter. Use a materialized read model when read scale and latency justify defined projection lag, replay, schema version, and authorization at hydration.
Restore is a workflow, not a file copy
If Orders restores to 10:00 and Payments to 10:07, cross-service workflows disagree. Recover to documented points, replay durable events, then reconcile order, payment, and ledger IDs before reopening writes. For example, a payment captured at 10:05 may still exist while the restored order database has forgotten its order. Compare durable business IDs and provider records before taking new action. Reconciliation should repair missing state or escalate ambiguity, not blindly repeat payment calls. Restoring each database successfully is only the first step toward restoring a consistent business workflow.
Schema and tenant isolation
The owning team uses expand/backfill/contract migrations and compatible event versions. Tenant keys, row policy, pool/queue budgets, and shard placement must remain enforceable in every derived view.
Reconciliation is a product capability
Compare business identities and amounts, classify missing/duplicate/out-of-order states, repair idempotently, retain evidence, and expose an operator queue for cases no algorithm can decide safely. Run reconciliation in bounded batches so it does not overload the owners it is inspecting. Store the detected discrepancy, supporting versions, chosen action, and repair result. Re-running the same repair should be harmless. Some differences are expected projection lag; use an age threshold and known workflow state before classifying them as corruption.
Migration states for one owned aggregate
SHARED_SOURCE
-> COPY_SNAPSHOT(checkpoint)
-> CATCH_UP_CDC(last_source_position)
-> SHADOW_READ(compare version + semantic fields)
-> NEW_WRITER(old store receives compatibility events)
-> NEW_SOURCE_OF_TRUTH(reconciliation SLO healthy)
-> REMOVE_OLD_ACCESS(after rollback window)
Invariant at every state:
one documented writer owns the aggregate version;
replayed commands/events are idempotent;
rollback never discards writes accepted by the new owner.| Recovery evidence | Required check | Do not reopen writes until |
|---|---|---|
| Event positions | Outbox/CDC high-water marks are known per owner | No unexplained source gap remains |
| Business reconciliation | Orders, captures, refunds, and ledger totals agree | Uncertain items are repaired or quarantined |
| Derived views | Projection version/checkpoint and tenant policy pass | Lag is inside the published consistency contract |
Takeaway
Own schema and writes per bounded context; compose reads with APIs or projections; coordinate money steps with sagas. Migrate ownership in phases, ship the outbox, and treat reconciliation as part of the product, not an afterthought.
Primary sources
- microservices.io: Database per Service: canonical pattern write-up.
- microservices.io: Shared Database. That is why sharing a schema couples teams.
- Transactional Outbox pattern: cross-service consistency without 2PC.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Give services enforceable data ownership while supporting cross-service reads, sagas, schema evolution, coordinated recovery, reconciliation, and tenant isolation.
Explain the decision
How should a UI query data owned by five services?
Read the reasoning
Use API composition for fresh low-fan-out queries or a versioned materialized read model for scale; do not join their private tables.
Apply it to a production scenario
Orders restore to 10:00 but payments restore to 10:07 after an incident. How do you recover?
Read one approach
Restore by documented consistency points where possible, replay durable events, reconcile business IDs and ledger state, and keep uncertain workflows quarantined.
Work through the numbers
A read model projects 12 million 800-byte entities with two indexes adding 50% and three replicas. Estimate storage.
Read the worked calculation
12M × 800 × 1.5 × 3 ≈ 43.2 GB before database/page/WAL overhead.
Reason through a failure
What is the ownership invariant?
Read the recovery reasoning
Only the owning service writes its authoritative data; integration paths are versioned contracts, and replay/reconciliation can rebuild every derived view.
Sources and further study
Continue with Availability Concepts.