Blob / Object Storage

How multipart and presigned URLs differ

A presigned request delegates a narrowly scoped operation, such as uploading to one chosen key before expiry; it does not mean the uploaded object is trustworthy. Record the expected owner, size policy, and processing state in application metadata. After completion, verify the resulting object and checksum where supported before marking the upload ready. Keep pending and rejected objects separate from content the application exposes publicly.

They solve different problems. Multipart means the service knows about "part 7 of 42": you retry individual slices and complete when all succeed. Presigned URLs are short-lived signed HTTPS requests: your backend vouches once; the browser or app uploads or downloads straight to object-storage endpoints.

Interview soundbite: we avoid proxying gigabytes through app servers. Multipart is for throughput and resume; a presigned PUT is for trust boundaries and cost.

Multipart upload

Multipart upload creates an upload session, stores individually numbered parts, and assembles the object only after a successful completion request. A client can retry one failed part without retransmitting the rest. Persist enough session information to resume safely, verify completion instead of assuming every part implies a finished object, and abort abandoned sessions so incomplete parts do not accumulate storage charges.

Use when objects exceed comfortable single-PUT limits or when networks drop mid-flight. Exact part-size rules vary by provider (minimum size except last part, maximum parts, and so on). Server-side initiation keeps IAM credentials off the client.

1. Initiate. Create a multipart session; receive an upload id and optionally a recommended part size.
2. Upload parts. PUT each numbered part in parallel; each returns an ETag (part identifier) you must retain. Retry only failed parts: a massive win versus restarting a multi-GB file.
3. Complete or abort. Send the ordered list of parts; the service concatenates into one object key. Abort unfinished uploads to avoid stranded billable storage. Lifecycle rules can garbage-collect stale multipart ghosts.

• More parts give more concurrency; balance client CPU and API rate limits.
• Use provider checksum headers when correctness of the assembled object matters.
• Persist upload id and completed parts so a reconnect can resume.

Presigned URLs

Your service signs a constrained request (method, bucket/key, headers, expiry). Anyone holding the URL can perform only that operation until it expires, so design the constraints deliberately narrow.

Follow with an event notification, webhook, or client callback so your domain model knows the object landed.

• Minutes to tens of minutes beats never-expires, limiting leaked-link damage.
• Sign exactly one key and verb; optional content-type and length constraints.
• Treat URLs like bearer tokens; never log query strings with secrets.
• Bytes bypass app autoscalers; you still pay PUTs and download egress.

Browser uploads of very large files often use multipart upload initiated server-side, then presigned URLs for each part (or a presigned POST policy for smaller files). You keep IAM credentials on the backend while clients stream slices in parallel.

Retention from those daily bytes

Using the ingest rates above: 5 TB/day of originals for 30 hot days keeps 150 TB hot. Keeping originals for 365 days adds about 1.8 PB total before replication. If derived playback files are 3 TB/day and kept hot for 90 days, add 270 TB. For logs at 200 GB/day compressed 5x, 13 months keeps about 15.6 TB compressed; seven-year archive keeps about 102 TB compressed.

A typical lifecycle policy timeline: Day 0–30 Standard (hot); Day 30–90 warm infrequent; Day 90–365 instant archive or cool; Day 365+ deep archive; Year 7+ delete per retention policy. Keep latest photo derivatives hot and expire orphaned previous versions after the restore window. Move upload originals warm after processing if playback uses derived objects, then archive the source after the compliance window. Logs stay hot 7–30 days for debugging, compressed analytics for 13 months, then archive or delete.

Replication

Same-region copies isolate accounts or aggregate logs. Cross-region async replication aids DR and locality: budget storage twice plus replication API and egress. Optional replication-time Service-Level Agreements (SLAs) exist as paid features; read the fine print. Replication PUTs, cross-region transfer, and destination storage all bill.

Performance patterns

Avoid designing key layouts from an assumed universal prefix limit. Some stores benefit from distributing requests across prefixes, while services such as S3 automatically partition and scale prefixes; verify the current service guidance and measure the workload before randomizing keys. Byte-range requests, multipart transfer, and CDN caching address different bottlenecks. Keep queryable ownership and lifecycle metadata in a database instead of treating object listing as a general-purpose index.

• Prefer ab/2026/05/user_id/object_id where ab is a hash prefix, to avoid a thundering herd.
• Avoid monotonically increasing prefixes such as 2026/05/16/... for all writes.
• Edge ingest onto the provider backbone can help distant uploads; benchmark versus direct PUT.
• Range GET enables parallel partial downloads for video players and big archives.
• Push-down filters over row or column files return fewer bytes than whole objects.

Migrating a bad prefix safely: dual-write new objects to the hashed layout, backfill old keys in batches, keep a redirect map from old key to new key, then expire old keys after readers and caches have moved. That avoids a risky big-bang rename, which is usually copy plus delete for every object.

Event-driven pipelines

An object-created notification means processing can begin, not that the video is already playable. Give each processing job a stable object version and transformation identifier, then make repeated notifications harmless. A worker writes derivatives to new keys and updates metadata to ready only when the required outputs exist. Failed jobs remain visible for retry or repair. Periodic reconciliation can find uploaded objects whose processing event was delayed or whose workflow stalled.

Object notifications fan out to queues or functions for thumbnails, virus scan, or warehouse ingestion, which decouples producers from processors. Pair with dead-letter queues and idempotent handlers: events can duplicate.

CDN in front

Serve public reads from an edge cache with restricted origin access; signed URLs or cookies protect paid streams. The origin bucket stays private.

• High cache hit ratios materially reduce origin GET cost and origin load.
• Origin access policies keep buckets non-public so only the edge can fetch.
• Modern edges can resize or transcode images on the fly.