SwadhyAI Notes

All System Design notes

System Design Economics

Cost Engineering

Imagine Instagram keeping every uploaded photo forever on hot SSD, with three replicas in every region "for safety." The feed still loads, and the invoice grows faster than monthly active users. Latency SLOs look fine; do not.

Capacity decisions have dollar signs. Cost engineering connects replication, retention, caching, and instance choice to cost per request, per GB, and .

Follow the money

1

Size the journey

QPS, storage growth, fan-out paths.

2

Cost per request, GB, or MAU.

3

Name top drivers

Compute, storage, egress, managed APIs.

4

Pick a lever

Retention, cache, async, right-size.

Try it: dial capacity economics before you scale

Numbers are illustrative USD-style estimates. Raise QPS, shift from reserved to on-demand, increase egress, or add replication to see which knob moves the monthly total fastest. Use this to prioritize architecture reviews, not as a quote.

Compute

$893/mo in this model.

Egress

$324/mo · CDN and region placement reduce this first.

Durable storage

$144/mo · replication × 2.

Monthly total
$1361
Compute
$893
Egress
$324
$/1M req
$37805.56

~$1,361/mo · compute $893 · egress $324 · storage $144 · ~$37805.56 per million requests. Turn the dials that dominate a typical API bill: compute shape, purchase option mix, egress, and storage replication.

Egress is architectural. Chatty mobile clients and cross-region fan-out show up here before CPU saturates.

Showback drives behavior. When teams see cost per service and per tenant, they right-size instances and cache aggressively.

What actually happens on the invoice

Once the whiteboard meets a real cloud bill, a design that meets 1M QPS but costs 10× per transaction loses to a simpler design that meets the SLO. Interview answers should tie caches, tiers, and to both latency and monthly burn, not only boxes on a whiteboard.

Compute is instances × $/hr × hours. Storage multiplies by and retention tiers. Cross-AZ and cross-region often dominate surprise bills: CDN byte hit ratio matters. Managed services add $/million requests, indexed logs, and metric series cardinality.

Track as features add fan-out. Over-provisioned DR regions and idle fleets are a silent tax; and autoscaling bounds belong next to budget alarms.

How to choose cost levers

After you see where the money goes, pick levers in this order so you do not optimize the wrong line item.

1. What are the top two cost drivers? Name them: storage, , compute, or managed APIs: before optimizing everything.

2. Can you cut spend without breaking SLO? Raise , shorten hot , move batch to , or absorb peaks with queues instead of peak-sized fleets.

3. Did you count multipliers? Indexes, backups, and multiply storage. Quoting instances alone understates the bill.

Simple rule of thumb

After sizing QPS and storage, estimate monthly cost and . Name one lever that reduces the top driver without missing latency or durability targets. Scale efficiency is a design requirement, not a FinOps afterthought.

In simple terms: architecture without economics is incomplete.

From problem to mechanism

The choice questions above become a repeatable model: demand to capacity to price to unit cost.

Problem. Latency SLOs are green, but monthly burn grows faster than MAU. Finance asks why a "small" media feature added seven figures of cloud spend.

Naive system. Size for peak QPS with generous headroom, keep three hot replicas forever, serve all bytes from origin, ignore cross-AZ and CDN byte-hit ratio. Invoice is "ops' problem."

Break it. A cache-key change bypasses CDN; origin doubles overnight. Migration replicas linger. Debug log retention explodes cardinality. Unit cost silently eats margin.

Mechanism. Build : demand → capacity → price → $/request and . Rank drivers (compute, storage×replication, egress, managed APIs). Apply levers: , , , , only when SLO/RPO still hold. Reconcile model to the bill.

Correctness invariant

The monthly model reconciles to the bill within an agreed tolerance and preserves required SLO, headroom, durability, and recovery constraints. A "saving" that only shifts cost or externalizes reliability risk does not count.

Levers that move cost

Move cold data only after defining access frequency, retrieval delay, and recovery requirements. A cheaper archive tier can add restore fees and latency during an incident. Run interruption-tolerant batch work on discounted interruptible capacity only when checkpoints and retries keep its deadline achievable. For each proposed saving, record the workload assumption and recheck the latency, durability, and recovery targets it must still meet.

  • : 30-day hot vs forever media on SSD.
  • : 90% vs 99% hits shrinks the origin fleet.
  • Async queues absorb peaks without sizing the whole fleet for max.
  • Compression and columnar formats cut analytics storage and scan cost.
  • for batch; on-demand for latency paths.

Worked example: egress dominates the invoice

With the mechanism in hand, run the same numbers once and reuse them (see Numbers to know) instead of restating every section.

Use the following illustrative inputs, not current provider quotes: 1.2 billion monthly requests, a peak of 3,000 requests per second, 30% capacity headroom, and a measured safe throughput of 180 requests per instance per second at $0.42/hour. The storage and egress calculation below uses 80 TiB of logical storage with three total copies and 150 TiB of monthly delivered bytes, priced per GiB for consistent binary units. Real invoices may use different units and already include internal storage redundancy.

Compute

ceil(3000×1.3/180)=22

22 × $0.42 × 730 ≈ $6,745

Storage

80TiB×3×1024GiB/TiB×$0.023

≈ $5,652 (before indexes/backups)

Egress

150TiB×1024GiB/TiB×$0.05

≈ $7,680: largest of the three

Unit cost

~$16.73 / M req

≈ $0.33/customer at 60K MAU on this subtotal

Numbers to know

  • Peak 3,000 RPS with 30% headroom → 22 instances at $0.42/hr ≈ $6,745/mo.
  • 80 TiB × 3 copies × 1024 GiB/TiB × $0.023/GiB-month ≈ $5,652 storage (before indexes/backups).
  • 150 TiB egress × 1024 GiB/TiB × $0.05/GiB ≈ $7,680: top driver in this model.
  • Subtotal ≈ $16.73 / M requests; ≈ $0.33 per customer at 60K MAU.
  • CDN byte-hit 40% → 60% saves ~30 TiB origin traffic before CDN fees.

If total delivered traffic is 150 TiB/month, a 40% CDN byte-hit ratio leaves 90 TiB at origin and a 60% hit ratio leaves 60 TiB: thirty TiB less origin traffic. That saving is priced at the origin-to-CDN or origin-delivery rate actually charged. CDN-to-user delivery still carries its own charge, so moving bytes to the edge does not automatically eliminate internet egress costs. Keep the two paths separate when comparing the new bill.

Unit economics worksheet

Divide the same reconciled subtotal by the relevant output. The example’s roughly $20,077 monthly subtotal divided by 1,200 million requests is about $16.73 per million requests; divided by 60,000 active customers it is about $0.335 per customer. State that support, engineering, and other services are excluded. Also measure cost per successful business operation, because retries can make cost per request look smaller while the useful work becomes more expensive.

Convert architecture choices into the same units so trade-offs are comparable. Formulas reuse the drivers above.

Compute

Compute cost is the sum of instance-hours at each effective rate, not always peak replica count multiplied by a full month. An always-on fleet of twenty-two instances at $0.42/hour and 730 hours costs about $6,745; a smaller baseline with temporary peak capacity changes that total. Autoscaling must become ready before demand and preserve failure headroom. Commitments can discount stable demand but are poor savings if paid capacity remains unused.

Storage

Storage cost depends on billable retained capacity in each tier, including the copies or redundancy the service actually charges separately. RF=3 normally means three total copies, not one primary plus three extra copies. A managed object-storage price may already include internal redundancy, so do not multiply that price again unless you intentionally maintain extra billed buckets or regions. Add snapshots, indexes, retained versions, requests, and retrieval fees explicitly.

Trace the path of each large byte flow: client delivery, origin-to-CDN misses, cross-zone service calls, cross-region replication, and backup copies may be billed differently. A cache can reduce database reads while increasing cross-zone traffic if placed poorly. Attribute costs by path and region before changing architecture, and include private service traffic rather than counting only bytes leaving the public API.

Managed services

$/million requests + indexed log GB + metric series count: cardinality explosions show up here.

≈ total_monthly_cost / monthly_active_users: track the trend as features add fan-out.

When the bill still bites

Even with a reconciled model (see Numbers to know), the wrong lever or missing ownership puts spend back on the rise.

Failure timeline: egress cost surprise

Day 0  Release changes thumbnail cache key; CDN miss rate spikes
Day 1  Origin egress +92%; traffic only +4%. Forecast alarm fires
Day 2  Team tunes instance size (wrong lever): bill still climbing
Day 3  FinOps attributes bytes to media path; rollback cache key
Day 4  Byte-hit recovers; temporary migration replicas found still billed
Day 5  Model updated: egress by path + CDN fee net case; owner assigned
Day 7  Unit cost back within band; alert on $/GB and forecast variance

Watch: cost/request and cost/MAU trends, egress by path, commitment coverage, idle DR capacity, log/metric cardinality, and forecast variance broken into usage vs price vs efficiency.

Still fails when…

  • Showback without ownership: nobody acts on unit-cost regressions.
  • Aggressive sampling cuts debug time into multi-hour incidents.
  • Spot reclaim correlates; on-demand catch-up costs more than savings.

Trade-offs

  • Cold archive savings vs read latency on rare restores.
  • Reserved discount vs flexibility when architecture changes.
  • Idle multi-AZ headroom looks wasteful until failover day.

Architecture as unit economics

What production needs to hold

A production design needs a way to recognize trouble and a rule it preserves while recovering. A trigger is the measurement or event that starts a response. An invariant is a statement that must remain true, such as never confirming the same reservation to two customers.

Trigger metric or condition

Build a cost model when spend grows faster than users or requests, egress and telemetry surprise the team, idle resilience capacity dominates, or a design choice lacks a measurable price/performance trade-off.

Correctness invariant

The monthly model reconciles to the bill within an agreed tolerance and preserves required SLO, headroom, durability, and recovery constraints; savings are not counted if they merely shift cost or externalize reliability risk.

From demand forecast to cost guardrail

Worked monthly capacity and sensitivity model

An API serves 1.2 billion requests/month, peaks at 3,000 requests/second, and provisions capacity 30% above peak demand. One instance safely handles 180 requests/second at the p99 SLO and costs $0.42/hour. For this hypothetical model, it stores 80 TiB of logical data in three separately billed copies at $0.023/GiB-month and sends 150 TiB/month at $0.05/GiB. Check real billing units and whether internal storage replication is already included before applying such a multiplier.

Compute

ceil(3,000 × 1.30 / 180) = 22 instances; 22 × $0.42 × 730 ≈ $6,745

The rounded fleet has 3,960 requests/second of capacity, leaving about 24.2% idle at the 3,000 peak. A requirement to keep 30% of provisioned capacity idle would instead need ceil(3,000 / (180 × 0.70)) = 24 instances. Add any separate failover or maintenance reserve explicitly.

Storage

80 TiB × 3 × 1,024 GiB/TiB × $0.023/GiB-month ≈ $5,652

Indexes, backups, compaction headroom, snapshots, and temporary migration copies belong in separate multipliers rather than being forgotten.

Egress

150 TiB × 1,024 GiB/TiB × $0.05/GiB ≈ $7,680

Egress is the largest of these three lines. A CDN or compression change should be modeled by byte-hit ratio, not request-hit ratio alone.

Unit cost

($20,077 modeled subtotal / 1.2B) × 1M ≈ $16.73/M requests

If 60K customers are active, the same subtotal is about $0.33/customer; allocate shared costs with a stated driver.

Decision

Sensitivity: a 20% traffic increase raises variable egress almost linearly, but compute changes only when the next capacity step is crossed. With 150 TiB total delivered traffic, improving byte-hit ratio from 40% to 60% reduces origin traffic from 90 to 60 TiB. Price that difference on the origin-to-CDN path; CDN-to-user bytes still incur their applicable charge.

How the mechanism holds

Follow how each part protects the rule above. The mechanism explains what the system does; the reasoning explains why that action is sufficient under the stated assumptions. If an assumption changes, the guarantee may change too.

Cost equations

Track cost/request, cost/successful journey, cost/active customer, storage/object, and telemetry/service. Separate fixed, stepwise, and variable components so forecasts respond correctly.

Why it holds: Model totals reconcile with tagged billing exports and usage meters within tolerance.

Egress and replication

Count client, cross-zone, cross-region, backup, cache-fill, and observability bytes. Replication factor multiplies storage and write I/O; global replication may add transfer and retained logs.

Why it holds: A data-flow byte ledger explains the top network billing lines.

Reserved, spot, and idle headroom

Cover stable baseline with commitments after sensitivity analysis; use spot for checkpointable work; keep on-demand for uncertainty. Price failover and maintenance headroom as reliability insurance.

Why it holds: Loss of spot and one zone still meets the declared degraded-mode capacity.

Attribution and showback

Tag ownership, environment, product, tenant tier, and cost center; allocate shared clusters by a defensible driver such as CPU-seconds, bytes, or requests. Showback informs before chargeback enforces.

Why it holds: Owners can trace a unit-cost regression to usage and architecture, not only an invoice category.

Cost incident: bill rises 45% in one week

1

Forecast alarm fires

Daily run rate departs from seasonally adjusted forecast while traffic grows only 4%.

Response: Break variance into price, usage, and efficiency; preserve reliability guardrails.

2

Egress owner is found

A release bypassed CDN for large thumbnails, doubling origin bytes.

Response: Rollback cache-key change, validate byte-hit ratio, and cap origin request/byte rates.

3

Secondary waste appears

Temporary migration replicas were never removed and logs use debug retention.

Response: Apply expiry owners and retention policy; never delete recovery data without RPO/audit check.

4

Model is corrected

Actual CDN and request-tier prices differ from the old flat assumptions.

Response: Version rate inputs, rerun sensitivity, and add variance alert ownership.

How to choose among alternatives

Reserved vs on-demand

Choose when: Commit the low-variance baseline; leave volatile growth flexible.

Caution: Over-commitment converts a discount into idle waste and limits architecture moves.

Spot vs reliable capacity

Choose when: Use spot for interruptible, checkpointed batch and replicas beyond the safety floor.

Caution: Correlated reclamation and queue catch-up must fit deadlines.

Optimize vs redesign

Choose when: Tune waste first; redesign when unit cost scales with the wrong dimension or dominates margin.

Caution: Engineer time, migration double-run, and operational complexity are costs too.

Running it day to day

Measurements tell you what is happening; alerts tell an operator when action is needed. A rollout plan limits how many users encounter a change before it is checked. A runbook records the steps to diagnose and recover from a known failure.

Measure

  • cost/request, journey, customer, GB, and service
  • utilization/headroom, commitment coverage, spot interruption, and idle rate
  • egress by path, replication/storage multipliers, and forecast variance

Alert

  • daily run-rate departs from forecast
  • unit cost rises without product/quality explanation
  • unowned resource, expired temporary capacity, or budget threshold

Roll out safely

  • baseline price and usage assumptions with owners/date
  • canary savings and verify SLO plus true net cost
  • roll out with automatic rollback on reliability or cost regression

Runbook

  • decompose variance into usage, price, and efficiency
  • stop accidental fan-out/retention while protecting recovery
  • attribute owner, correct model, and verify durable unit-cost improvement

Practice scenario

Estimate monthly cost and cost per million requests for the worked API, then reduce it 20% without violating a 30% headroom requirement.

Questions worth asking next

  • What changes under 20% demand growth?
  • How do reservations and spot mix?
  • Who pays for shared observability?

A solid answer sketch

Model peak compute, replicated/indexed storage, every egress path, managed requests, and support headroom. Rank drivers; here egress is largest, so improve compression and byte-hit ratio, tier retention, then commit only stable compute baseline. Attribute shared costs with measurable usage.

Common mistakes

  • • using average QPS to size latency-critical compute
  • • ignoring indexes, backups, migration copies, or cross-zone bytes
  • • calling idle DR waste without valuing RTO and capacity during failover

Last reviewed: 6 August 2026

Choosing and explaining cost engineering

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

After sizing QPS and storage, estimate monthly cost and cost per user.

Name the top 2 cost drivers and one lever to reduce them without breaking SLO.

Over-provisioning is a hidden tax: tie autoscaling bounds to budget alarms.

When it belongs

Design compares managed vs self-hosted.

Retention or replication multipliers dominate storage.

FinOps or unit economics question in senior interviews.

How it fails

Cross-region replication doubles egress silently.

Log/metric cardinality explosion.

Idle over-provisioned DR region year-round.

Trade-offs to weigh

Cheaper cold storage vs read latency.

Aggressive sampling vs debug time in incidents.

Reserved capacity savings vs flexibility.

Common misunderstandings

Quoting instances without storage and egress.

Ignoring index and backup multiplication.

Scaling team faster than revenue per MAU.

Wrap-up

You have sized the drivers, walked the egress surprise, and seen the worksheet. Use these exercises to lock the habit of naming the top driver before picking a lever.

Exercises

  1. Cut 20% without breaking 30% headroom. Using the worked model (see Numbers to know), name the top driver and one lever. What happens to unit cost under +20% demand?
  2. Egress vs compute debate. A PM wants bigger instances "for safety." Finance wants CDN. Pick one change for the next sprint and justify with $/M requests.
  3. DR region looks idle. Finance wants to delete the warm standby. How do you price RTO insurance vs true waste? What metric proves the standby is unused vs readiness?

Topic-specific references

Key takeaway

Architecture without economics is incomplete. Size for peak with explicit multipliers, then ask whether each multiplier is worth the invoice, and instrument cost per critical journey alongside latency.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Model cost per request/customer, egress, replication, reservations/spot, idle headroom, attribution, and sensitivity without trading away SLOs blindly.

Explain the decision

Why is average infrastructure utilization not a complete cost metric?

Read the reasoning

Headroom covers peaks and failures; unit economics must include availability target, egress, storage copies, licenses, people, and waste attribution.

Apply it to a production scenario

A cheaper region adds 80 ms and doubles egress. How do you decide?

Read one approach

Compare user/conversion impact, residency, resilience, per-request egress, reserved pricing, and reversible experiment results—not compute price alone.

Work through the numbers

A service costs $180,000/month and serves 9 billion requests. Cost per million requests?

Read the worked calculation

$20 per million requests, before shared attribution and margin allocation.

Reason through a failure

What must a cost optimization preserve?

Read the recovery reasoning

Correctness, security, recovery capacity, and declared SLO/error-budget policy; savings are invalid if risk is merely hidden.

Sources and further study

Continue with API Compatibility & Consumer Contracts.