Delivery

UPLOAD PATH (write):
  Client → Presigned Upload URL → Object Storage (S3)
  → Backend never touches the file bytes
  → S3 triggers event → Processing Queue

PROCESSING PATH (async):
  Queue → Worker picks up job
  → Download original from S3
  → Transcode: 1080p, 720p, 480p, thumbnail
  → Upload variants back to S3
  → Update metadata DB: "video_123 ready, 3 variants"

DELIVERY PATH (read):
  Client requests video_123
  → Backend generates presigned CDN URL (720p variant)
  → Client fetches directly from CDN edge
  → CDN cache HIT? → serve from edge (10ms)
  → CDN cache MISS? → fetch from S3 origin, cache, serve

Key insight: the backend NEVER serves file bytes.
  Upload: client → S3 directly (presigned upload URL)
  Download: client → CDN directly (presigned download URL)
  Backend only handles: auth, metadata, URL generation

DOWNLOAD FLOW:

  1. Client: GET /api/videos/123
  2. Backend:
     → Verify auth (is this user allowed to access video 123?)
     → Generate presigned URL:
       https://cdn.example.com/videos/123/720p.mp4
         ?X-Amz-Algorithm=AWS4-HMAC-SHA256
         &X-Amz-Credential=AKIA.../20250115/us-east-1/s3/aws4_request
         &X-Amz-Date=20250115T120000Z
         &X-Amz-Expires=3600          ← valid for 1 hour
         &X-Amz-Signature=abc123...   ← cryptographic signature
     → Return URL to client

  3. Client: fetches video directly from CDN using the presigned URL
     → CDN edge has it cached? → serve immediately (10ms)
     → CDN cache miss? → CDN fetches from S3, caches, serves

  Backend involvement: ~5ms (auth + URL generation)
  File transfer: 0 bytes through backend ✅

UPLOAD FLOW:

  1. Client: POST /api/uploads (request upload URL)
  2. Backend:
     → Generate presigned upload URL for S3
     → Return URL + upload instructions
  3. Client: PUT file directly to S3 using presigned URL
     → 500MB video goes straight to S3, not through backend
  4. S3 triggers event → processing pipeline starts

Upload event triggers pipeline:

  S3 Event: "video_123.mp4 uploaded"
       │
       ▼
  Message Queue (SQS / Kafka)
       │
       ▼
  Transcoding Workers (auto-scaling)
       │
       ├→ Worker 1: video_123 → 1080p.mp4 (H.264, 5 Mbps)
       ├→ Worker 2: video_123 → 720p.mp4  (H.264, 2.5 Mbps)
       ├→ Worker 3: video_123 → 480p.mp4  (H.264, 1 Mbps)
       ├→ Worker 4: video_123 → thumbnail.jpg (frame at 2s)
       └→ Worker 5: video_123 → HLS manifest (adaptive streaming)
       │
       ▼
  Upload variants to S3:
    s3://videos/123/1080p.mp4
    s3://videos/123/720p.mp4
    s3://videos/123/480p.mp4
    s3://videos/123/thumbnail.jpg
    s3://videos/123/manifest.m3u8
       │
       ▼
  Update metadata DB:
    video_123: status = "ready", variants = [1080p, 720p, 480p]

Key principles:
  → Always async (never block the upload response)
  → Queue decouples upload from processing
  → Workers auto-scale based on queue depth
  → Each variant is an independent job (parallelizable)

User A uploads: cat_meme.jpg (2.3 MB)

  1. Compute hash: SHA-256(file_bytes) = "a1b2c3d4e5f6..."
  2. Check storage: does object "a1b2c3d4e5f6..." exist?
     → NO: upload file to S3 as "a1b2c3d4e5f6..."
     → Store metadata: { user: A, filename: "cat_meme.jpg", hash: "a1b2c3..." }

User B uploads: funny_cat.jpg (same image, different filename)

  1. Compute hash: SHA-256(file_bytes) = "a1b2c3d4e5f6..." (same!)
  2. Check storage: does object "a1b2c3d4e5f6..." exist?
     → YES: skip upload (file already stored)
     → Store metadata: { user: B, filename: "funny_cat.jpg", hash: "a1b2c3..." }

Result:
  Storage: 1 copy of the file (2.3 MB, not 4.6 MB)
  Metadata: 2 entries pointing to the same hash
  Savings: 50% storage reduction for this file

At scale (Dropbox, Google Drive):
  Millions of users upload the same popular files
  Deduplication saves 30-60% of total storage
  That's petabytes of savings = millions of dollars

Without client-side dedup:
  Client uploads 500MB video → server hashes → duplicate found
  → 500MB wasted bandwidth, upload took 2 minutes for nothing

With client-side dedup:
  1. Client computes: SHA-256(file) = "a1b2c3..."
  2. Client asks server: POST /api/uploads/check { hash: "a1b2c3..." }
  3. Server checks: hash exists? → YES
  4. Server responds: { "exists": true, "file_id": "file_789" }
  5. Client skips upload entirely — just links to existing file
  → 0 bytes uploaded, instant "upload complete"

Dropbox uses this: "instant upload" for files that already exist
in any user's storage. The file never leaves the client's machine.

UPLOAD:
  1. Client computes SHA-256 of video file
  2. Client: POST /api/uploads { hash: "a1b2c3...", size: 524MB, type: "video/mp4" }
  3. Server checks dedup: hash exists?
     → YES: skip upload, return existing file_id (instant!)
     → NO: generate presigned S3 upload URL
  4. Client uploads directly to S3 via presigned URL
     → 524MB goes to S3, not through backend
  5. S3 triggers event → message sent to transcoding queue

PROCESSING (async):
  6. Transcoding worker picks up job
  7. Downloads original from S3
  8. Transcodes: 1080p, 720p, 480p, thumbnail, HLS manifest
  9. Uploads all variants to S3:
     s3://videos/{hash}/1080p.mp4
     s3://videos/{hash}/720p.mp4
     s3://videos/{hash}/480p.mp4
     s3://videos/{hash}/thumb.jpg
     s3://videos/{hash}/manifest.m3u8
  10. Updates DB: video status = "ready"

DELIVERY:
  11. Client: GET /api/videos/456
  12. Backend: auth check → generate presigned CDN URL for 720p
  13. Client fetches from CDN:
      → CDN edge in user's region has it cached? → 10ms ✅
      → Cache miss? → CDN fetches from S3, caches, serves → 200ms
  14. Video player uses HLS manifest for adaptive streaming
      → Switches between 480p/720p/1080p based on bandwidth

DEDUPLICATION IN ACTION:
  User B uploads the same video (different title):
  → SHA-256 matches → skip upload entirely
  → Create new metadata entry pointing to same hash
  → Storage: 1 copy serves both users