System Design · Storage Internals

Document Store Internals

A catalog item is a nested object: title, per-locale copy, an array of variants, and a blob of attributes that differ by category. A store writes that object as one unit, indexes the fields you query, and routes it by a . Flexibility is real. Magic is not: under the JSON you still have pages, journals, planners, and hot shards.

This lesson follows bytes on the path MongoDB-style engines actually take: , a , indexes (including arrays), replica-set durability, and why unbounded blows up a document. It is not “NoSQL versus SQL” at slogan level—that belongs with access-pattern notes.

One document write

Router sends the doc to one primary.

Durability first; cache second.

Every index is another write.

w:majority is not w:1.

What actually happens

The client speaks JSON. The engine stores (or a similar binary form) so nested fields are skippable by length prefixes. A is a heap of those blobs plus a primary key index (often _id) and optional secondary indexes. The storage engine pages those blobs like any other row store: cache, eviction, checkpoints.

Clustered deployments split two jobs. A copies the oplog for durability and failover. Sharding splits collections by shard key across several replica sets. The router is a scatter/gather coordinator, not a query genius: a filter that does not include the shard key may touch every shard.

Modeling is the remaining half. makes the common read one round-trip. Referencing keeps documents small and shared data unique. The failure mode is an unbounded nested array that forces a rewrite of a multi-megabyte BSON blob on every comment.

On-disk document shape

plaintext

Logical JSON:
  { "_id": "sku_9", "title": "Mug", "tags": ["kitchen","sale"],
    "variants": [{ "sku": "a", "price": 9 }, { "sku": "b", "price": 11 }] }

BSON-ish layout (simplified):
  [doc_length][type][_id][value][type][title][len][bytes]
  [type=array][tags][len][elem0][elem1]...
  [type=array][variants][len][subdocs...]

Implications:
  - skip unknown fields without a JSON parse
  - updating one nested field may rewrite the whole document
    if the engine cannot pad/in-place grow
  - 16 MB-class size caps exist for a reason: one doc = one unit
  - field names repeat on disk; short names still help at huge scale

In-place updates need slack space. Engines historically used padding; modern WiredTiger relocates documents that grow past their page slot and updates the id index. Rapidly growing documents therefore create write amplification even when the API looks like a tiny $push.

Write path

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart TD
    C[Client insert/update] --> R[Router mongos]
    R --> P["Primary of owning shard"]
    P --> J[Journal / WiredTiger log]
    J --> D["Update document page + indexes"]
    D --> O[Append oplog entry]
    O --> Ack{"Write concern met?"}
    Ack -->|yes| OK[Ack client]
    O --> S[Secondaries apply oplog]

Unsharded: skip the router, still hit a primary. Sharded: the must be present on inserts. A missing or unhashed monotonic key is how one shard takes all writes while the others idle.

is the durability dial. w:1 can lose an acknowledged write if the primary dies before the oplog reaches a majority (depending on journaling and failover). w:majority waits; latency follows the slowest needed secondary plus journal flush. Product people who want “fast writes” and “never lose an order” are asking for two settings.

plaintext

INSERT product (sharded on { shopId: 1, sku: 1 }):
  1. hash/range locate chunk → shard S
  2. primary(S): write WT journal
  3. insert BSON into collection table
  4. update _id index + each secondary index
     (multikey: one entry per array element)
  5. append oplog
  6. wait writeConcern
  7. secondaries apply in oplog order

Cost sketch:
  1 document, 6 indexes, tags[20] on a multikey index
  → 1 doc write + ~25 index maint operations
  not “one JSON POST”

Read path and the planner

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart TD
    Q[Query + filter] --> Pl[Query planner]
    Pl --> Idx{Usable index?}
    Idx -->|no| Scan["Collection scan on targeted shards"]
    Idx -->|yes| Ix[Index walk → document ids]
    Ix --> Fetch[Fetch documents from WiredTiger]
    Scan --> Filt[Apply remaining predicates]
    Fetch --> Filt
    Filt --> Scatter{Multiple shards?}
    Scatter -->|yes| Merge[Merge / sort / skip on router]
    Scatter -->|no| Out[Return batch]
    Merge --> Out

A yields document ids (or clustered keys), then a fetch. Covered queries that project only indexed fields skip the fetch—the same covering-index idea as SQL. A on an array field is how “find products with tag kitchen” stays an index walk instead of a collection scan.

Scatter-gather is the tax for queries that cannot target chunks. Ten shards and a filter on an unindexed field is ten collection scans plus a merge on the router. That is not horizontal scale; that is ten times the work with extra network.

Query shape	What the engine should do	Failure if you skip design
By _id	Primary index, one shard if _id is the shard key or hashed	Broadcast _id lookup if _id is not the shard key
By shopId + sku	Targeted shard + compound index	Fine—this is the modeled path
By tag only	Multikey index, often scatter	Scatter + COLLSCAN without the index
Sort + skip for infinite scroll	Indexed sort; prefer range on last id	skip(N) walks N keys every page

Cache, working set, and WiredTiger

The JSON API hides a page cache. The is documents you keep touching plus the index pages those queries walk. When it exceeds the , eviction fights the application: dirty pages must flush, clean pages die, p99 jumps. This is the same cliff as an undersized InnoDB buffer pool.

Indexes are often larger than people sketch. A multikey index on a 20-element array is roughly 20× the key cardinality for that field. Compound indexes duplicate prefixes. Count RAM for indexes first, then documents, then “the cluster has disk so we are fine.”

plaintext

Rough working-set budget:
  documents hot per minute × avg BSON size
  + all index pages for those query shapes
  + oplog window you want on disk (time, not “default”)

If cache = 32 GB and hot BSON + indexes ≈ 48 GB:
  expect page-in latency, not in-memory microseconds

Design move:
  project fewer fields, bound arrays, drop unused indexes,
  split cold history into another collection/cluster

Embedding, references, and document growth

wins for “product with 8 variants” and “user with 3 addresses.” It loses for “post with unbounded comments” and “inbox with unbounded messages.” Hard size caps, rewrite amplification, and lock/latch duration all scale with document bytes, not with how cute the nested JSON looks.

Embed

1:1 or small 1:N, same lifecycle, always fetched together, rarely updated independently. One round-trip, transactional with the parent on one shard.

Reference

Unbounded N, shared entities, independent queries, or a different shard key. Extra round-trips or application joins; often a second collection with its own indexes.

Hybrid patterns exist: embed a preview (last 20 comments) and keep the rest in a child collection. That is an explicit denormalization with a refresh rule, not a free lunch.

Replication, sharding, and change streams

A is the HA unit. Elections promote a secondary; clients must retry. Sharding is orthogonal: each shard is typically its own replica set. Chunk migrator jobs move ranges; during a move you can see extra latency and orphan documents if you ignore balancer health.

Downstream caches and search indexes should follow the , not dual-write from the app if you can avoid it. Dual-write diverges. The oplog is already ordered. Resume tokens and idempotent consumers are part of the design, the same as any CDC pipeline.

plaintext

Read your own write (common product bug):
  writeConcern: majority
  then read from a secondary  →  may miss the write
  Fix: read from primary, or use causal/session consistency

Shard-key change:
  not a live ALTER in older generations; plan a new collection
  and backfill. Treat the key as a one-way bet.

Balancer:
  hot chunk splits help until the key itself is hot
  (one shopId taking 40% of writes cannot be split away)

Worked micro-estimate

plaintext

10M products, avg BSON 4 KB → 40 GB documents
indexes:
  _id            ~ 10M × 20 B ≈ 0.2 GB
  {shopId, sku}  ~ 0.4 GB
  tags multikey  avg 8 tags → ~ 10M × 8 × 20 B ≈ 1.6 GB
  text-ish extra indexes    ≈ 2 GB
→ indexes ~ 4+ GB, plus WiredTiger overhead

Hot shop = 5% of SKUs but 50% of QPS
  if shard key is sku hashed: writes spread, shop queries scatter
  if shard key is shopId: shop queries target one shard, that shop hotspots

Choose:
  shopId hashed + shopId+sku index  OR
  compound shard key {shopId, sku} with careful chunking
Measure chunk distribution; do not assume “auto-sharding” heals a hotspot.

Trade-offs to weigh

Flexible documents versus implicit schema drift

You can ship a new field without a migration. You can also accumulate three spellings of the same field and break every index. Use validators, version fields, and explicit migrations for money-shaped attributes even in a document store.

Embedding versus references

Embedding buys one-read locality and single-document atomicity. References buy unbounded growth and independent indexing. Unbounded embeds hit size caps and rewrite the whole parent on every child append.

Index coverage versus write amplification

Each query shape you want fast wants an index. Each index slows writes and inflates the working set. Audit unused indexes; they are not free insurance.

Write concern versus latency

Majority concern plus journaling is the honest setting for orders. Session caches can use weaker settings. Do not mix them on one collection without naming the loss window in the runbook.

Targeted shards versus scatter-gather

A shard key that matches the primary query path keeps work on one replica set. Secondary access patterns will scatter. Design for the top QPS path, then add indexes or derived collections for the rest—do not expect one key to make every find() cheap.

Choosing and explaining document store internals

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

A document store persists nested BSON-like records, maintains indexes, and uses a page cache and journal underneath the JSON API.

Writes hit one primary (and one shard when keyed); each secondary index and array element multiplies the physical work.

Reads are cheap when they use an index and the shard key; otherwise the router scatters collection scans.

When it belongs

Entities vary by tenant or category and are usually read as one object.

You must justify embedding versus a child collection, or a shard key versus a hotspot.

Latency issues look like “Mongo is slow” and you need working set / planner language.

How it fails

Hot shard from a monotonic or celebrity shard key.

Collection scans and skip-based pagination at scale.

Unbounded embedded arrays hitting size limits and rewrite amplification.

w:1 acknowledgements lost on primary failover.

Trade-offs to weigh

Schema flexibility versus validators and index stability.

Single-document atomicity versus multi-document transactions and joins.

Replica-set read scale versus replication lag.

Common misunderstandings

Calling it schemaless as if indexes and shard keys were optional.

Assuming auto-sharding removes hotspots.

Treating secondary reads as linearizable.

Confusing document stores with analytical columnar files.

Wrap-up

Document engines are row stores with nested records and a rich planner, not bags of JSON on disk. Size the working set, pick a shard key for the dominant path, bound embeds, and set write concern on purpose. The internals story is journals, indexes, oplogs, and routers—the same vocabulary as other storage engines, with a different record shape.

Primary sources

Key takeaway

One document is one atomic nested record. Indexes, shard keys, and cache size decide whether that convenience stays fast. Embed the bounded; reference the rest; never confuse a flexible field list with a free query planner.