Transaction Patterns

Place an order (3 services involved):

  1. Order Service    → Create order record
  2. Payment Service  → Charge customer's card
  3. Inventory Service → Reserve items

Happy path: all 3 succeed → order confirmed ✅

Failure scenarios:
  Step 1 succeeds, Step 2 fails
    → Order exists but payment failed
    → Must cancel the order (compensate Step 1)

  Steps 1-2 succeed, Step 3 fails
    → Order exists, payment charged, but items unavailable
    → Must refund payment (compensate Step 2)
    → Must cancel order (compensate Step 1)

  Step 2 succeeds but Step 2's response is lost
    → Did payment go through? We don't know.
    → Retry? Might charge twice.
    → Don't retry? Might lose the charge.

This is why distributed transactions are hard.

Coordinator manages the transaction across 3 services:

PHASE 1 — PREPARE (voting):
  Coordinator → Order Service:     "Can you commit?"  → "YES" ✅
  Coordinator → Payment Service:   "Can you commit?"  → "YES" ✅
  Coordinator → Inventory Service: "Can you commit?"  → "YES" ✅

  All voted YES → proceed to Phase 2

PHASE 2 — COMMIT:
  Coordinator → Order Service:     "COMMIT"  → committed ✅
  Coordinator → Payment Service:   "COMMIT"  → committed ✅
  Coordinator → Inventory Service: "COMMIT"  → committed ✅

  Transaction complete.

FAILURE SCENARIO:
  Phase 1: Inventory votes "NO" (out of stock)
  Coordinator → ALL services: "ABORT"
  Order Service rolls back, Payment Service rolls back.
  → All-or-nothing, like a local transaction.

COORDINATOR FAILURE:
  Coordinator crashes after Phase 1 (all voted YES) but before Phase 2
  → All services are BLOCKED. They voted YES but don't know if they
     should commit or abort. They hold locks and wait.
  → This is the fatal flaw of 2PC.

HAPPY PATH:
  Order Service: Create order → publish "OrderCreated"
       ↓
  Payment Service: hears "OrderCreated" → charge card
       → publish "PaymentCompleted"
       ↓
  Inventory Service: hears "PaymentCompleted" → reserve items
       → publish "ItemsReserved"
       ↓
  Order Service: hears "ItemsReserved" → mark order as confirmed ✅

FAILURE PATH (inventory fails):
  Order Service: Create order → publish "OrderCreated"
       ↓
  Payment Service: charge card → publish "PaymentCompleted"
       ↓
  Inventory Service: items out of stock!
       → publish "ReservationFailed"
       ↓
  Payment Service: hears "ReservationFailed" → REFUND payment
       → publish "PaymentRefunded"
       ↓
  Order Service: hears "PaymentRefunded" → CANCEL order

Each step has a compensating action:
  Create order    ↔  Cancel order
  Charge payment  ↔  Refund payment
  Reserve items   ↔  Release reservation

Orchestrator (Order Saga):

HAPPY PATH:
  Step 1: Tell Order Service → "Create order"        → OK ✅
  Step 2: Tell Payment Service → "Charge $99.99"     → OK ✅
  Step 3: Tell Inventory Service → "Reserve 2 items"  → OK ✅
  Step 4: Tell Order Service → "Confirm order"        → DONE ✅

FAILURE PATH (Step 3 fails):
  Step 1: Tell Order Service → "Create order"        → OK ✅
  Step 2: Tell Payment Service → "Charge $99.99"     → OK ✅
  Step 3: Tell Inventory Service → "Reserve 2 items"  → FAILED ❌
  
  Compensate Step 2: Tell Payment Service → "Refund $99.99"  → OK
  Compensate Step 1: Tell Order Service → "Cancel order"     → OK
  
  Saga failed. All steps compensated. System is consistent.

The orchestrator:
  → Knows the full flow (steps + compensations)
  → Tracks the current state of the saga
  → Decides what to do on success or failure
  → Is the single source of truth for the saga's progress

THE PROBLEM:
  // Step 1: Write to database
  db.insert("orders", order);  // ✅ succeeds
  
  // Step 2: Publish event to Kafka
  kafka.publish("OrderCreated", order);  // ❌ fails (network error)
  
  // Result: order exists in DB but no event was published
  // Payment service never hears about the order
  // Customer is confused: "I placed an order but nothing happened"

  Reversing the order doesn't help:
  kafka.publish("OrderCreated", order);  // ✅ succeeds
  db.insert("orders", order);            // ❌ fails
  // Event published but order doesn't exist in DB!

THE SOLUTION — OUTBOX PATTERN:
  // Single database transaction:
  BEGIN;
    INSERT INTO orders (...) VALUES (...);
    INSERT INTO outbox (event_type, payload) VALUES ('OrderCreated', ...);
  COMMIT;
  // Both succeed or both fail — atomic!

  // Background worker (separate process):
  // Reads outbox table → publishes to Kafka → marks as published
  SELECT * FROM outbox WHERE published = false;
  kafka.publish(event);
  UPDATE outbox SET published = true WHERE id = ...;

1. Application writes:
   BEGIN TRANSACTION;
     INSERT INTO orders (id, user_id, total) VALUES ('ord-123', 42, 99.99);
     INSERT INTO outbox (id, event_type, payload, published)
       VALUES ('evt-456', 'OrderCreated', '{"orderId":"ord-123",...}', false);
   COMMIT;
   → Both writes are atomic. If either fails, both roll back.

2. Outbox relay (background worker):
   Every 100ms:
     SELECT * FROM outbox WHERE published = false ORDER BY created_at LIMIT 100;
     For each event:
       kafka.publish(event.event_type, event.payload);
       UPDATE outbox SET published = true WHERE id = event.id;

3. Consumers receive the event:
   Payment Service → hears "OrderCreated" → charges card
   Inventory Service → hears "OrderCreated" → reserves items

4. Cleanup:
   DELETE FROM outbox WHERE published = true AND created_at < NOW() - INTERVAL '7 days';

Order Saga Orchestrator:
  │
  ├── Step 1: Order Service → Create order
  │   └── Outbox → "OrderCreated" → Kafka
  │
  ├── Step 2: Payment Service → Charge card
  │   └── Outbox → "PaymentCompleted" → Kafka
  │
  ├── Step 3: Inventory Service → Reserve items
  │   └── Outbox → "ItemsReserved" → Kafka
  │
  └── Step 4: Order Service → Confirm order

Failure at Step 3:
  Orchestrator → Payment Service: "Refund" (compensate Step 2)
  Orchestrator → Order Service: "Cancel" (compensate Step 1)

Each service:
  → Uses Outbox pattern (atomic DB write + event)
  → Is idempotent (safe to retry)
  → Has compensating actions defined