Task Queuing

Client → API → Queue → Worker(s) → Result Store

1. Client: "Process this video" (POST /uploads)
2. API: creates task, enqueues it, returns job_id
   Response: { "job_id": "job-abc-123", "status": "queued" }
   → Client gets response in ~50ms

3. Queue: holds the task until a worker picks it up
   Tasks: [job-abc-123, job-def-456, job-ghi-789, ...]

4. Worker: dequeues task, processes it (3 minutes)
   → Transcode video, generate thumbnails, run moderation

5. Result Store: worker writes result
   → Update DB: job-abc-123 status = "completed", output_url = "..."

6. Client polls or gets notified:
   GET /jobs/job-abc-123 → { "status": "completed", "url": "..." }
   OR: WebSocket/push notification when done

Job: {
  id: "job-abc-123",
  type: "transcode_video",
  payload: { video_id: "vid-42", format: "720p" },
  status: "queued",        ← tracked by the queue
  retries: 0,              ← auto-retry on failure
  max_retries: 3,
  scheduled_at: "2024-06-15T10:00:00Z",  ← delayed execution
  created_at: "2024-06-15T09:55:00Z"
}

Lifecycle:
  queued → running → completed ✅
                   → failed → retry → running → completed ✅
                                     → failed (max retries) → dead ❌ (→ DLQ)

Features:
  ✅ Status tracking (query job state anytime)
  ✅ Retry with backoff (automatic on failure)
  ✅ Scheduling (run at a specific time)
  ✅ Priority (urgent jobs first)
  ✅ Timeout (kill jobs that run too long)

Examples: Sidekiq (Ruby), Celery (Python), BullMQ (Node.js), Temporal

Message: {
  topic: "order-events",
  key: "user-42",
  value: { "event": "OrderPlaced", "order_id": "ord-123", "total": 99.99 },
  timestamp: "2024-06-15T10:00:00Z"
}

Lifecycle:
  published → delivered → acknowledged
  (no status tracking, no retry management by the queue itself)

Features:
  ✅ Pub/Sub (one message → many consumers)
  ✅ Ordering (within a partition)
  ✅ Replay (consumers can re-read old messages)
  ✅ High throughput (millions of messages/sec)
  ❌ No job status tracking
  ❌ No built-in retry/scheduling (consumer handles this)

Examples: Kafka, SQS, RabbitMQ, Redis Streams

Three priority levels:

HIGH priority queue:   [password-reset, payment-alert, security-warning]
MEDIUM priority queue: [order-confirmation, shipping-update]
LOW priority queue:    [weekly-report, data-export, cleanup-job]

Worker behavior:
  1. Check HIGH queue → any tasks? → process
  2. If HIGH is empty → check MEDIUM → process
  3. If MEDIUM is empty → check LOW → process

Result:
  Password reset email: sent in 2 seconds ✅
  Order confirmation: sent in 30 seconds ✅
  Weekly report: generated in 2 hours ✅

Implementation options:
  Option A: Separate queues per priority (simplest)
    Workers poll HIGH first, then MEDIUM, then LOW

  Option B: Single queue with priority field (heap-based)
    Queue sorts by priority internally
    Higher priority tasks dequeued first

  Option C: Weighted fair queuing
    HIGH gets 60% of worker capacity
    MEDIUM gets 30%
    LOW gets 10%
    → Prevents starvation of low-priority tasks

Task: "Send welcome email to user-42"

Attempt 1: Email service is down → FAILED → retry in 10s
Attempt 2: Email service still down → FAILED → retry in 30s
Attempt 3: Email service returns 500 → FAILED → retry in 60s
Attempt 4 (max retries reached): FAILED → move to DLQ

Main Queue:  [task-1, task-2, task-3, ...]  ← healthy tasks
Dead-Letter: [failed-task-A, failed-task-B]  ← failed tasks

DLQ contains:
  {
    original_task: { type: "send_email", user_id: 42 },
    failure_reason: "Email service returned 500",
    attempts: 4,
    first_failed_at: "2024-06-15T10:00:00Z",
    last_failed_at: "2024-06-15T10:01:30Z"
  }

What happens next:
  1. Alert: monitoring detects DLQ depth > 0 → page on-call engineer
  2. Investigate: engineer checks failure reason → email service was down
  3. Fix: email service is restored
  4. Replay: move tasks from DLQ back to main queue → processed successfully

Upload API
  │
  ├── Store video in S3
  ├── Create job in queue
  └── Return job_id to client (200ms)

Job Queue (BullMQ / Celery):
  HIGH:     [paid-user-videos]
  MEDIUM:   [free-user-videos]
  CRITICAL: [content-moderation]

Workers:
  Transcode workers (10x):  HIGH → MEDIUM queue
  Thumbnail workers (5x):   all priorities
  Moderation workers (3x):  CRITICAL queue only

Failure handling:
  Retry: 3 attempts, exponential backoff (10s, 30s, 90s)
  DLQ: after 3 failures → dead-letter queue
  Alert: DLQ depth > 0 → page on-call

Status tracking:
  GET /jobs/job-123 → { status: "running", progress: 65% }
  → Client shows progress bar
  → On completion: push notification + status update