Distributed File Systems

Files are split into large chunks so the master stores millions of chunk records instead of billions of tiny block pointers. Clients cache metadata briefly, then talk to chunkservers for bulk I/O. Heartbeats carry disk free space and chunk inventories; the master treats location maps as reconstructible soft state in the GFS design, which simplifies master recovery.

The consistency model is deliberately coarser than local POSIX. Concurrent writers, record appends, and replica failure produce defined regions that may be defined, undefined, or consistent-but-not-identical depending on the paper's rules. Batch analytics tolerate that better than databases do—which is why these systems excel at large sequential workloads.

Suppose 10 PB cluster, 64 MiB chunks, 3× replication:
  unique data chunks ≈ 10 PB / 64 MiB ≈ 164 million
  with 3 replicas: ~492 million chunk copies on disks
  master metadata (rough): path entries + chunk map
    often tens of bytes to hundreds per chunk/file
  → metadata fits in master RAM at this grain

If chunks were 4 KiB like a local FS block:
  ~2.7 trillion blocks → master RAM explodes
  client would thrash metadata RPCs

Large chunks also:
  + amortize TCP setup and seek costs for sequential scans
  − make partial overwrites and tiny-file workloads awkward
  − cause imbalance if a few huge files dominate

Tiny-file workloads are a known weak spot: each small file still costs namespace metadata and often underfills a chunk. Systems add packing, HAR-like archives, or push small objects to a different store. Do not force a DFS into a latency-sensitive random 1 KB read product without redesigning the grain.

Clients pick among replica locations using network distance (same rack preferred) and observed errors. Short-circuit reads in HDFS let a task read from its local DataNode when the scheduler placed compute near data—an explicit co-design with MapReduce / YARN-style locality.

Mutation serialization is per chunk via a lease. The primary orders writes, forwards to secondaries, and only then acks. Data flows in a pipeline so the client is not uploading three full copies sequentially. Version numbers advance so a returning node with old bytes becomes a stale replica and is dropped from the serving set until recopied.

Lease sketch:
  master.grant(chunkId, primary, replicas, timeout)
  primary renews before expiry while writers are active
  if primary dies:
    lease expires → master picks new primary from up-to-date replicas
    stale replicas (lower version) are excluded

Record append (GFS):
  at-least-once client retry may create duplicates
  applications use checksums / opaque records to ignore padding / dups
  → designed for producers that can tolerate that contract

rack awareness policies vary, but the invariant is: replica sets should survive the failure domains you declare (disk, node, rack). re-replication is continuous background work. After a rack outage, thousands of chunks may be under-replicated simultaneously; masters prioritize based on urgency (only one copy left) and popularity.

The master logs namespace mutations to an edit log / operation log and periodically checkpoints an image. Classic HDFS secondary namenode language confuses interviews: it is primarily a checkpoint helper, not a transparent hot standby. Production HA adds a shared journal (or quorum journal nodes) plus an active/standby pair with fencing.

MapReduce and lake-house pipelines lean on that pattern: tasks write temporary paths, then atomically rename into a committed directory the next job lists. Concurrent random updates inside a multi-gigabyte file are possible but rare in the sweet spot—the lease primary becomes a hotspot, and readers must understand which byte ranges are defined.

When a product needs mutable objects with HTTP semantics, an object store (or a KV/blob layer) is usually the clearer abstraction. Keep the DFS where hierarchical bulk datasets and compute locality dominate.

Common publish pattern:
  write  /tmp/job_42/part-0000
  ...
  rename /tmp/job_42  →  /warehouse/table/dt=2026-04-10/

Readers listing the date partition either see nothing or the full set
(assuming rename of the directory is atomic in the namespace).

Avoid:
  many clients overwriting overlapping ranges of the same open file
  without an external coordination story

Interviews often blur these. Say which API and consistency contract you need first. Many modern lakes use object storage under table formats; the GFS paper still teaches the control-plane / data-plane split cleanly.