Delivery Guarantees

Producer → Broker (Kafka, SQS, RabbitMQ) → Consumer(s)

Delivery guarantee applies at each hop:
  Producer → Broker:  Did the broker receive the message?
  Broker → Consumer:  Did the consumer process the message?

Failures can happen at any point:
  1. Producer sends, network drops it       → message lost
  2. Broker receives, crashes before storing → message lost
  3. Broker delivers, consumer crashes       → message not processed
  4. Consumer processes, ACK is lost         → broker retries → duplicate
  5. Network delivers message twice          → duplicate processing

Producer → Send message → Done (no retry)

Scenario 1 (success):
  Producer sends → Broker receives → Consumer processes ✅
  Message delivered once. Perfect.

Scenario 2 (failure):
  Producer sends → Network drops message → Lost forever ❌
  No retry. Producer doesn't know it was lost.

Scenario 3 (partial failure):
  Producer sends → Broker receives → Consumer crashes before processing
  Broker doesn't retry → Message effectively lost ❌

Internal mechanism:
  Fire-and-forget. No ACK tracking. No retry logic.
  UDP-style: send and hope for the best.

Producer → Send message → Wait for ACK → Retry if no ACK

Scenario 1 (success):
  Producer sends → Broker ACKs → Done ✅
  Message delivered once.

Scenario 2 (retry success):
  Producer sends → Network drops message → No ACK
  Producer retries → Broker receives → ACKs → Done ✅
  Message delivered once (retry worked).

Scenario 3 (duplicate!):
  Producer sends → Broker receives → Processes → Sends ACK
  ACK is lost in network → Producer thinks it failed
  Producer retries → Broker receives AGAIN → Processes AGAIN ⚠️
  Message delivered TWICE. Consumer must handle the duplicate.

Internal mechanism:
  Retry with exponential backoff until ACK received.
  Consumer must ACK after processing (not before).
  If consumer crashes after processing but before ACK → redelivery.

Problem: order message delivered twice → two orders created

Solution 1 — Idempotency key:
  Message: { idempotency_key: "order-abc-123", user: 42, total: 99.99 }
  Consumer: 
    if (db.exists("processed_keys", "order-abc-123")) → skip
    else → process order, store "order-abc-123" in processed_keys

Solution 2 — Database constraint:
  INSERT INTO orders (idempotency_key, user_id, total)
  VALUES ('order-abc-123', 42, 99.99)
  ON CONFLICT (idempotency_key) DO NOTHING;
  → Second insert silently ignored. No duplicate order.

Solution 3 — Natural idempotency:
  "SET user balance to $500" is naturally idempotent.
  Running it twice produces the same result.
  "ADD $100 to balance" is NOT idempotent — running twice adds $200.

Technique: At-least-once + Idempotency + Transactional processing

Step 1 — At-least-once delivery:
  Broker retries until consumer ACKs. No message lost.

Step 2 — Deduplication:
  Each message has a unique ID.
  Consumer checks: "Have I seen this ID before?"
  If yes → skip. If no → process.

Step 3 — Transactional processing:
  Process the message AND record the message ID in ONE transaction.
  BEGIN;
    INSERT INTO orders (...) VALUES (...);
    INSERT INTO processed_messages (id) VALUES ('msg-abc-123');
  COMMIT;
  → If the transaction fails, neither the order nor the ID is recorded.
  → On retry, the ID check prevents reprocessing.

Kafka's "exactly-once semantics" (EOS):
  Producer: idempotent producer (dedup at broker level)
  Consumer: read-process-write in a Kafka transaction
  → Offset commit + output write are atomic
  → If consumer crashes, it resumes from the last committed offset
  → No duplicate processing (transactional guarantee)

Kafka Partition (ordered log):
  Offset: [0]  [1]  [2]  [3]  [4]  [5]  [6]  [7]
  Data:   msg0 msg1 msg2 msg3 msg4 msg5 msg6 msg7

Consumer A reads:
  Start at offset 0 → process msg0 → commit offset 1
  Read offset 1 → process msg1 → commit offset 2
  Read offset 2 → process msg2 → commit offset 3
  (crash!)
  
  Restart → resume from last committed offset (3)
  Read offset 3 → process msg3 → continue...
  → No messages lost, no messages skipped

Consumer B (independent):
  Start at offset 0 → reads the same messages independently
  → Multiple consumers can read the same log at different speeds

Topic: "orders" with 4 partitions

Partition 0: [order-1] [order-5] [order-9]   → ordered within P0
Partition 1: [order-2] [order-6] [order-10]  → ordered within P1
Partition 2: [order-3] [order-7] [order-11]  → ordered within P2
Partition 3: [order-4] [order-8] [order-12]  → ordered within P3

Ordering guarantee:
  ✅ order-1 is processed before order-5 (same partition)
  ❌ order-1 might be processed after order-2 (different partitions)

How to ensure ordering for a specific entity:
  Partition key = user_id
  All orders for user_42 go to the same partition
  → user_42's orders are always processed in order
  → Different users' orders can be processed in parallel

Producer publishes: "OrderPlaced" event

                    ┌──→ Payment Service    (charge the card)
                    │
  OrderPlaced ──→ Broker ──→ Inventory Service (reduce stock)
                    │
                    ├──→ Notification Service (send confirmation email)
                    │
                    └──→ Analytics Service   (track conversion)

Each subscriber:
  → Receives the same event independently
  → Processes at its own pace
  → Has its own delivery guarantee
  → Can fail without affecting others

Kafka implementation:
  Topic: "order-events"
  Consumer Group A: Payment Service (1 instance)
  Consumer Group B: Inventory Service (3 instances)
  Consumer Group C: Notification Service (2 instances)
  Consumer Group D: Analytics Service (5 instances)
  → Each group gets every message. Within a group, messages are distributed.

Order API → Kafka (at-least-once, acks=all)
  │
  │ Topic: "order-events", partition key: user_id
  │
  ├── Payment Service     [at-least-once + idempotent]
  │   → Check dedup table → charge card → record order_id
  │
  ├── Inventory Service   [at-least-once + idempotent]
  │   → Check dedup → decrement stock → record order_id
  │
  ├── Notification Service [at-least-once]
  │   → Send email (duplicate email is annoying but not harmful)
  │
  └── Analytics Service    [at-most-once]
      → Increment counters (losing 0.1% is fine)

Ordering: guaranteed per user (same partition)
Fan-out: each service is an independent consumer group
Offset tracking: each service commits its own offset