Storage Patterns

UPLOAD FLOW:
  Client → Upload Service → Object Storage (S3)
                          → Metadata DB (PostgreSQL)

  1. Client uploads file to Upload Service
  2. Upload Service stores file in S3 → gets back a key (file-id)
  3. Upload Service writes metadata to DB:
     { id: "file-123", key: "uploads/file-123.jpg",
       size: 2048576, type: "image/jpeg", owner: "user-42",
       url: "https://cdn.example.com/uploads/file-123.jpg" }

DOWNLOAD FLOW:
  Client → CDN → Object Storage (S3)

  1. Client requests file URL from API
  2. API returns CDN URL (or pre-signed S3 URL)
  3. Client downloads directly from CDN/S3
  → Your servers never touch the file bytes on download

KEY PRINCIPLE:
  Metadata (small, queryable) → Database
  File bytes (large, opaque)  → Object Storage
  Delivery (fast, global)     → CDN

Each object:
  {
    key:      "uploads/user-42/photo-abc123.jpg"   (unique identifier)
    data:     <2.1 MB of binary image data>        (the actual file)
    metadata: {
      content-type: "image/jpeg",
      size: 2148576,
      uploaded-at: "2024-06-15T10:30:00Z",
      custom-tags: { "user-id": "42", "album": "vacation" }
    }
  }

Key features:
  → Flat namespace: "uploads/user-42/photo.jpg" is a KEY, not a path
     (the "/" is just a character in the key, not a directory separator)
  → Immutable: objects are written once, read many times
     (to "update" a file, you write a new object with the same key)
  → Versioning: optionally keep all versions of an object
  → Lifecycle policies: auto-delete after 90 days, move to cold storage

Without pre-signed URLs (bad):
  Client → Upload to your server → Server uploads to S3
  Problem: your server handles all file bytes → bandwidth bottleneck

With pre-signed URLs (good):
  1. Client asks your API: "I want to upload a photo"
  2. API generates a pre-signed S3 URL (valid for 15 minutes):
     https://bucket.s3.amazonaws.com/uploads/photo-abc.jpg
       ?X-Amz-Signature=abc123...&X-Amz-Expires=900
  3. Client uploads DIRECTLY to S3 using the pre-signed URL
  4. Client notifies your API: "Upload complete, key = photo-abc.jpg"
  5. API writes metadata to DB

Benefits:
  → Your servers never touch the file bytes
  → S3 handles the upload bandwidth (infinite capacity)
  → Pre-signed URL expires → no permanent public access
  → Works for downloads too (private files with temporary access)

File: vacation-video.mp4 (2 GB)
Chunk size: 100 MB → 20 chunks

Step 1 — Initiate upload:
  POST /uploads/initiate
  → Server returns upload_id: "upload-xyz-789"

Step 2 — Upload chunks (can be parallel):
  PUT /uploads/upload-xyz-789/chunk/1  → 100 MB → ✅
  PUT /uploads/upload-xyz-789/chunk/2  → 100 MB → ✅
  PUT /uploads/upload-xyz-789/chunk/3  → 100 MB → ❌ (network error)
  PUT /uploads/upload-xyz-789/chunk/3  → 100 MB → ✅ (retry only chunk 3)
  PUT /uploads/upload-xyz-789/chunk/4  → 100 MB → ✅
  ... (chunks 5-20 uploaded in parallel)

Step 3 — Complete upload:
  POST /uploads/upload-xyz-789/complete
  → Server reassembles chunks into final file
  → Stores in S3 as one object
  → Returns file URL

Resume after app crash:
  GET /uploads/upload-xyz-789/status
  → "Chunks 1-12 received, 13-20 missing"
  → Client resumes from chunk 13 (not from scratch)

Upload flow:
  1. Client computes hash: SHA-256(file) → "a1b2c3d4e5f6..."
  2. Client asks server: "Do you have hash a1b2c3d4e5f6?"
  3a. Server: "YES" → skip upload (file already exists!) → instant ✅
  3b. Server: "NO"  → upload the file, store with key = hash

Storage:
  Key: "a1b2c3d4e5f6..."
  Value: <file bytes>

Deduplication:
  User A uploads vacation.jpg → hash = "abc123" → stored
  User B uploads the same photo → hash = "abc123" → already exists!
  → No duplicate storage. Both users reference the same object.
  → 1 copy stored, 2 metadata entries pointing to it.

Integrity verification:
  Download file → compute SHA-256 → compare with stored hash
  Match → file is intact ✅
  Mismatch → file is corrupted ❌ (re-download or alert)

UPLOAD:
  Client → SHA-256(file) → API: "hash exists?"
    YES → Add reference in DB → Done (instant, no upload)
    NO  → API: initiate multipart upload
        → S3 returns pre-signed URLs per chunk
        → Client uploads chunks directly to S3 (parallel)
        → Client: "upload complete"
        → API: complete multipart upload in S3
        → API: INSERT metadata in PostgreSQL

DOWNLOAD:
  Client → API: "give me file-123"
  API → Generate pre-signed CDN URL (expires in 1 hour)
  Client → CDN edge → S3 origin (on cache miss)
  → Your servers handle 0 bytes of file data

DEDUPLICATION:
  10B files, 50 PB raw → with dedup: ~35 PB actual storage
  → 30% storage savings = millions of dollars saved

STORAGE TIERS:
  Hot (accessed in last 30 days):  S3 Standard
  Warm (30-90 days):               S3 Infrequent Access
  Cold (>90 days):                 S3 Glacier
  → Lifecycle policies auto-transition files between tiers