System Design Storage
File vs Block vs Object
A payment checkout writes ledger rows that must not lose a 4 KB page. An Instagram upload drops a video that will mostly be read, never patched. A design team shares a project folder with locks and paths. Same word: "storage": three different contracts.
Latency and mutability drive the split: for in-place database disks, for shared paths, for durable blobs over APIs.
Decision questions
In-place random writes?
Yes → start with block volumes.
Shared path tree?
Yes → file/NFS-style shares.
Whole-object replace OK?
Yes → object storage + CDN.
Need more than one?
Common: block for DB, object for media.
What actually happens
exposes fixed-size blocks. The guest OS or database decides what those blocks mean. You care about , latency, and snapshots. Cloud volumes such as are the usual building block for stateful VMs.
keeps the folder metaphor: often via . So apps that expect paths and locks can keep working. stores key + metadata + body with semantics and higher API latency, but excellent scale for media and archives.
Price rows later are illustrative public list tiers. Verify current regional pricing before budgeting.
Try it: choose file, block, or object storage
Each workload states mutability, sharing, and payload shape. Match the storage contract before comparing price sheets.
Payment ledger database
A relational primary store must rewrite 8 KB pages inside transactions and fsync often.
- In-place random writes?
- Yes
- Shared directory tree?
- No
- Huge immutable payloads?
- No
- Scenario
- 1 of 4
- Your pick
- None
- Expert pick
- Hidden
Block
Fixed blocks, low latency, snapshots; guest OS or DB owns layout.
File
Hierarchy, locks, and POSIX paths for shared editing.
Object
Flat keys, HTTP APIs, immutability, and CDN-friendly reads.
Pick a storage class, then reveal the expert pick to compare reasoning.
Block first for databases. Random in-place writes and snapshot semantics belong on volumes the engine controls.
File shares for path semantics. When multiple clients expect directories and locks, object key prefixes are a poor substitute.
Object for large immutable bytes. Media, backups, and analytics landing zones favor whole-object PUT and CDN reads over POSIX or block mounts.
How to choose storage type
1. Does the workload mutate small regions in place? Databases and boot disks want block. Do not put your primary OLTP store on object APIs.
2. Do multiple clients need a shared tree? Use file storage when path semantics and locks matter. Object prefixes are not a filesystem rename.
3. Are bytes mostly write-once, read-many? Prefer object storage for media, backups, and lake files: usually with a CDN in front for public reads.
Simple rule of thumb
Default hybrid: block volumes for databases, object storage for user media and analytics files, file shares only when an app truly needs shared paths. Pick from access pattern first; optimize cost second.
Three paradigms
Block storage exposes an addressable sequence of blocks that a filesystem or database organizes. File storage exposes paths, directories, and file operations managed by a filesystem. Object storage exposes complete objects by key through an API. The application-visible interface is the main distinction; implementations may all use replication, disks, and network services internally. Choose according to required operations, not the physical medium alone.
Latency and mutability drive the split: databases want in-place 4KB writes; archives want whole-object replaces with erasure-coded durability.
| Feature | File | Block | Object |
|---|---|---|---|
| Abstraction | Paths and directories | Fixed-size blocks | Blob + metadata + key |
| Interface | Shared filesystem API | Remote block protocol | REST (GET/PUT…) |
| Typical latency | Low ms | Sub-ms to low ms | Tens of ms (API hop) |
| Random access | Yes | Yes | Whole object (range reads supported) |
| Mutability | In place | In place | Replace object version |
| Cost trend | Higher $/GB (managed shared file) | Mid (per GB + input/output operations per second (IOPS)) | Lowest $/GB for cold data |
Beginner mental model
File = shared folder
/team/images/logo.png
Apps navigate paths, directories, permissions, and locks.
Block = raw disk
The OS/database decides what those blocks mean.
Object = blob with key
key: videos/42/720p.m3u8
metadata: type, checksum, owner
body: bytes
Clients GET/PUT whole objects or byte ranges over APIs.
Failure and recovery math
Restoring bytes takes time even when a snapshot is available. Copying 1 TB at a sustained 200 MB per second has a lower bound of about 5,000 seconds, or 83 minutes, before provisioning, replay, verification, and application startup. A snapshot may be crash-consistent without coordinating every application buffer, so recovery may also require logs. Test the full restoration path against the promised recovery time.
Recovery design asks how many bytes must be restored and how fast the system can safely read them back. Capacity alone does not answer outage time.
Block restore
Restore a 10 TB database volume from snapshot at 500 MB/s: about 5.6 hours before replaying logs and warming caches.
File failover
Namespace failover must recover locks, open handles, and metadata servers; tiny-file trees can slow recovery.
Object restore
Version restore or bulk undelete may be request-rate bound; archive retrieval can add minutes or hours before bytes flow.
Formula
restore_time = restored_bytes / safe_restore_bandwidth. Add time for consistency checks, index rebuilds, write-ahead log replay, and application validation.
Storage choice flow
Choose the abstraction that matches the application contract first, then optimize cost, lifecycle, and replication around that choice.
Workload
Low-latency random writes?
Yes
Block volume
Snapshots and replicas
No → shared path semantics?
Continue decision tree below
Shared path semantics?
Yes
File storage
Directory and lock scaling
No → large immutable blobs?
Continue decision tree below
Large immutable blobs?
Yes
Object storage
Lifecycle and Content Delivery Network
No
Revisit access pattern
Choose when apps truly need shared filesystem semantics or thousands of small randomized writes across the same tree. Otherwise object plus compute mounts often suffices cheaper.
Shared network filesystems
A shared filesystem lets several clients use one namespace, but concurrent writes require the filesystem’s locking and consistency rules to match application assumptions. A local-file application may block unexpectedly when the remote server is slow. File count, directory lookups, metadata updates, and lock contention can become bottlenecks before data bandwidth does. Test the actual access pattern, including many small files and concurrent writers.
Concurrent mounts from many VMs, lock semantics preserved. Throughput tiers scale with spend. Typical legacy CMS or shared config workloads.
Distributed analytics filesystems
Large-block sequential analytics filesystem co-located with compute clusters. Cloud-native stacks increasingly read columnar files straight from object storage instead.
Typical fits
- Shared config volumes across autoscaled VMs
- Filesystem-required machine learning (ML) stacks often complemented by high-throughput scratch caches
- Legacy CMS uploads expecting local filesystem paths
Watch metadata hotspots when millions of tiny files pile into one directory.
One provisioned volume usually attaches to one instance at a time (exceptions exist for clustered filesystems). You manage snapshots for durability beyond a single Availability Zone (AZ).
Block volume SKUs (illustrative)
| Type | Input/output operations per second (IOPS) profile | Throughput hint | Rough $/GB-mo |
|---|---|---|---|
| General purpose Solid State Drive (SSD) | Baseline + burst tuning | Configurable MB/s | ~0.08 |
| Provisioned I/O SSD | Provisioned IOPS | High | Higher |
| Throughput Hard Disk Drive (HDD) | HDD throughput | Sequential heavy | Lower $/GB |
Best for
Online transaction processing (OLTP) databases, boot disks, latency-sensitive journal files, and append-only database logs on durable volumes.
Limits
Availability Zone (AZ)-scoped durability until replicated via snapshots or mirrors; vertical scaling via larger volumes or striping.
Capacity math by storage type
Capacity, bandwidth, IOPS, and latency are separate limits. A database performing 20,000 random 4 KiB operations per second transfers about 78 MiB per second, which is modest bandwidth but substantial IOPS demand. A backup stream can transfer hundreds of megabytes per second using relatively few large operations. Two workloads with the same stored bytes can therefore need very different storage performance.
Same user base, different storage shape: block is sized for IOPS and recovery, file for namespace and metadata, object for payload, retention, and derivatives.
Profile photos on object storage
1M users x 2 MB original + 200 KB derivatives ≈ 2.2 TB payload. Metadata at 1 KB/object adds about 1 GB per million objects.
Media upload service
100K uploads/day x 50 MB ≈ 5 TB/day raw. Add derived renditions and retention policy before estimating monthly object storage.
Database block volume
If the active dataset is 1 TB and indexes add 40%, start near 1.4 TB plus write-ahead logs, snapshots, and growth headroom.
Shared file namespace
Millions of tiny files can bottleneck on directory metadata before bytes. Budget inode/metadata operations, not just GB.
Object vs block vs file answer
Object wins cheap immutable bytes and lifecycle. Block wins low-latency random writes and database semantics. File wins shared directories and legacy path contracts.
| Type | Size these first | Hidden pressure |
|---|---|---|
| Block | Capacity, IOPS, throughput, snapshots, restore time | Write-ahead logs, replica lag, snapshot growth, headroom |
| File | Bytes, file count, directory fan-out, metadata operations/s | Locks, inode pressure, hot directories, namespace failover |
| Object | Payload, object count, metadata, requests, lifecycle, egress | Small-object overhead, listings, retrieval fees, CDN miss rate |
Managed tiering, versioning, lifecycle transitions, and Identity and Access Management (IAM) policies differ per cloud, but the mental model stays: PUT/GET by key, durability via replication or erasure coding, CDN (Content Delivery Network) for delivery.
HTTP verbs (object APIs)
| Tier | Illustrative $/GB-mo | Use |
|---|---|---|
| Hot standard | ~0.02 | Frequent reads |
| Infrequent access | Lower storage, retrieval fees | Backups, Disaster Recovery (DR) copies |
| Deep archive | Fraction of a cent | Compliance tapes replacement |
Decision cheat sheet
| Workload | Pick | Why |
|---|---|---|
| Relational database data directory | Block | Random writes + fsync semantics |
| User assets + CDN | Object | Cost + HTTP caching layers |
| Shared filesystem dataset | File | Multi attach semantics |
| Data lake files | Object | Columnar file economics |
| VM root volume | Block | Bootloader expects block device |
When not to use each type
The fastest interview answer is often saying what you reject. Pick by access semantics first, then tune cost and lifecycle.
Avoid object storage when
The app needs low-latency random in-place writes, file locks, or single-digit millisecond database fsync behavior.
Avoid shared file storage when
The workload is massive immutable public media and path semantics are not required; object storage plus CDN is usually simpler.
Avoid block storage when
The data is a cheap long-term archive; snapshots and provisioned performance can make it expensive.
Picking storage under pressure
Start from access shape: random writes and fsync, shared path semantics, or cheap immutable blobs. Then mention durability, latency, and scaling limits.
Photos and profile images
Object storage for originals and derivatives, metadata database (DB) for ownership/state, CDN for reads, lifecycle/versioning for retention.
Relational database primary disk
Block storage because the database needs low-latency random I/O, page cache, WAL/fsync behavior, and snapshots or replicas for recovery.
Shared files for legacy enterprise app
File storage if the app expects filesystem paths, directory listings, and multiple clients mounting the same namespace.
Security logs kept seven years
Object storage with time partitions, immutable retention/lock if required, lifecycle to archive, and query through lake or warehouse tools.
Service-Level Agreement (SLA) wording that sounds senior
"I would separate latency Service-Level Objectives (SLOs) from durability goals: block/file choices optimize hot I/O semantics, while object storage gives cheaper multi-AZ durability but needs metadata, retries, lifecycle rules, and restore drills."
Hybrid patterns
A common design keeps database pages on block storage, shared legacy assets on a filesystem where required, and immutable uploads in object storage. The database stores the object key and ownership metadata rather than the large media body. This avoids forcing whole-object APIs to behave like random page writes or forcing application instances to serve every video byte. Each boundary still needs a consistent creation and deletion workflow.
Tier hot/warm/cold automatically; cache object origins behind CDNs or in-memory clusters; on-prem object stores can preserve the object API programming model.
Tiered lifecycle
Logs hot on SSD-backed volumes, aged partitions compressed to object cold tiers with retrieval SLAs.
Cache façade
Authoritative bytes live in object storage; PoPs or in-memory clusters absorb flash crowds.
Object-compatible on-prem
On-prem object stores reuse familiar tooling while satisfying data residency requirements.
Rough cost illustration (100 TB month)
Capacity is only one line item. Real estimates also include operations, transfer, retrieval, snapshots, recovery drills, and support for peak restore.
| Tier | Order of magnitude | Cost shape |
|---|---|---|
| General-purpose block | Thousands of dollars / month | Capacity + IOPS + snapshots + recovery headroom |
| Managed shared file | Often highest $/GB | Capacity + throughput tier + metadata operations |
| Object standard | Low thousands | Capacity + PUT/GET requests + transfer + metadata |
| Deep archive object tier | Hundreds | Low capacity + retrieval fee + restore delay |
Degraded modes and storage drills
Storage choice sets recovery and failure behavior. Block, file, and object each degrade differently when an AZ, volume, or metadata path fails.
Block volume AZ loss
Fail over the database replica or restore from snapshot. Degrade to read-only if write promotion would violate RPO.
Shared file namespace slow
Legacy apps may hang on locks. Cap concurrent mounts, shed batch jobs, and avoid expanding file storage for new media workloads.
Object tier retrieval delay
Archive restores can take hours. Keep hot copies for paths that need interactive access; show explicit restore-pending UX.
Snapshot / backup gap
Capacity without restore practice is false comfort. Schedule timed restores and measure RTO against the SLA you quote.
Test and drill notes
- Restore a block snapshot into a scratch instance and verify application boot plus checksum of a known dataset.
- Object archive drill: restore a cold prefix and record time-to-first-byte and cost.
- Rule of thumb: pick storage by access semantics first, then attach durability, lifecycle, and restore drill evidence.
Storage type estimation matrix
Choose by access semantics first, then size the bottleneck dimension: IOPS for block, metadata ops for file, object count + lifecycle for object.
| Type | Size by | When not to use |
|---|---|---|
| Block | Capacity, IOPS, snapshots, restore time | Cheap archive or shared read-heavy media |
| File | File count, directory ops, lock rate | Massive immutable public CDN media |
| Object | Objects, metadata, request rate, egress | Low-latency random in-place writes |
Takeaway
Block for databases and VMs, object for media and lake files, file only when shared filesystem semantics are unavoidable. Automate lifecycle transitions so prod datasets cool down without tickets.
Primary sources
- Ghemawat et al.: Google File System: chunked file storage.
- AWS Storage Services Overview: file vs block vs object.
- SNIA: Cloud Data Management Interface: storage interface taxonomy.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Choose file, block, or object storage from access semantics, latency, IOPS, sharing, consistency, snapshots, durability, and recovery needs.
Explain the decision
Why is object storage a poor drop-in POSIX filesystem?
Read the reasoning
Object APIs replace whole objects and lack the same low-latency random writes, locking, directory, rename, and shared-mount semantics.
Apply it to a production scenario
A database uses a shared multi-attach block volume across nodes. What must be proven?
Read one approach
The filesystem/database stack supports coordinated multi-writer access; otherwise use one writer with replication or a clustered filesystem.
Work through the numbers
A workload needs 60,000 random IOPS and each volume supplies 8,000 at target latency. How many volumes before redundancy?
Read the worked calculation
ceil(60,000 / 8,000) = 8 volumes; throughput, queue depth, striping, and failure rebuild may require more.
Reason through a failure
What must a snapshot strategy prove?
Read the recovery reasoning
Application-consistent restore, not merely crash-consistent blocks; regularly test dependencies, keys, metadata, and recovery time.
Sources and further study
Continue with Data Representation and Storage Concepts.