Coordination Patterns

Leader Election — Avoiding the Herd Effect:

Candidates: [candidate-0001, candidate-0002, candidate-0003, candidate-0004]

Watch chain (each watches the one before it):
  candidate-0001 (LEADER) ← watched by candidate-0002
  candidate-0002          ← watched by candidate-0003
  candidate-0003          ← watched by candidate-0004
  candidate-0004

When candidate-0001 (leader) crashes:
  1. /election/candidate-0001 is deleted (ephemeral)
  2. ONLY candidate-0002 gets notified (it was watching 0001)
  3. candidate-0002 checks: "Am I lowest?" → YES → becomes leader
  4. candidate-0003 and 0004 are NOT disturbed

Compare to naive approach (everyone watches the leader):
  1. Leader crashes
  2. ALL candidates get notified simultaneously
  3. ALL candidates call getChildren at once (thundering herd)
  4. Server overwhelmed with N simultaneous requests
  
The "watch previous" pattern: O(1) notifications per failure
The "watch leader" pattern: O(N) notifications per failure

// Leader election implementation (pseudocode)
async function participateInElection(zk: ZooKeeper) {
  // Step 1: Create our candidate node
  const myNode = await zk.create(
    "/election/candidate-",
    myInfo,
    EPHEMERAL_SEQUENTIAL
  );
  
  const mySeq = extractSequence(myNode); // e.g., 0000000003
  
  async function checkLeadership() {
    // Step 2: Get all candidates
    const children = await zk.getChildren("/election");
    const sorted = children.sort(); // sort by sequence number
    
    if (sorted[0] === basename(myNode)) {
      // Step 3: I have the lowest number — I'm the leader!
      onBecomeLeader();
      return;
    }
    
    // Step 4: Watch the node just before me
    const myIndex = sorted.indexOf(basename(myNode));
    const previousNode = sorted[myIndex - 1];
    
    const exists = await zk.exists(
      `/election/${previousNode}`,
      (event) => {
        // Step 5: Previous node deleted — re-check
        checkLeadership();
      }
    );
    
    // Edge case: previous node already gone before watch set
    if (!exists) {
      checkLeadership();
    }
  }
  
  await checkLeadership();
}

Fair Distributed Lock (Queue-Based):

Lock path: /locks/resource-X

Acquire lock:
  1. Create EPHEMERAL_SEQUENTIAL: /locks/resource-X/lock-0000000005
  2. getChildren("/locks/resource-X") → [lock-0003, lock-0004, lock-0005]
  3. Am I the lowest? (lock-0003 is lowest, not me)
  4. Watch lock-0004 (the one just before me)
  5. Wait for notification...

  When lock-0004 is deleted:
  6. getChildren again → [lock-0003, lock-0005]
  7. Still not lowest (lock-0003 exists)
  8. Watch lock-0003
  
  When lock-0003 is deleted:
  9. getChildren → [lock-0005]
  10. I'm the lowest! → LOCK ACQUIRED ✅

Release lock:
  11. Delete /locks/resource-X/lock-0005
  12. Next waiter (lock-0006) gets notified → acquires lock

Crash recovery (automatic):
  If lock holder crashes → ephemeral node deleted → next waiter proceeds
  No deadlocks from dead processes!

Read-Write Lock:
  Writers: /locks/resource-X/write-0000000001
  Readers: /locks/resource-X/read-0000000002
  
  Writer acquires: must be lowest of ALL nodes (read + write)
  Reader acquires: must be lowest of all WRITE nodes before it
  Multiple readers can hold the lock simultaneously

Service Discovery Pattern:

ZooKeeper namespace:
/services
├── payment
│   ├── instance-1    data: {"host":"10.0.1.5","port":8080,"version":"2.1"}
│   ├── instance-2    data: {"host":"10.0.1.6","port":8080,"version":"2.1"}
│   └── instance-3    data: {"host":"10.0.2.1","port":8080,"version":"2.2"}
├── inventory
│   ├── instance-1    data: {"host":"10.0.1.7","port":9090}
│   └── instance-2    data: {"host":"10.0.2.2","port":9090}
└── notification
    └── instance-1    data: {"host":"10.0.1.8","port":7070}

All instance nodes are EPHEMERAL:
  - Instance crashes → node auto-deleted → consumers notified
  - Instance gracefully shuts down → deletes node → consumers notified
  - No stale registrations (unlike DNS-based discovery with TTLs)

Consumer pattern:
  1. getChildren("/services/payment", watch=true)
     → ["instance-1", "instance-2", "instance-3"]
  2. For each instance: getData to get host:port
  3. Build connection pool to all instances
  4. On watch event: re-read, update connection pool

Advantages over DNS-based discovery:
  ✅ Instant notification (vs DNS TTL delay)
  ✅ No stale entries (ephemeral = auto-cleanup)
  ✅ Rich metadata in node data (version, capabilities)
  ✅ Consistent view (all consumers see same state)

Configuration Management Pattern:

Structure:
/config
├── database        data: {"host":"db.prod","port":5432,"pool_size":20}
├── feature-flags   data: {"dark_mode":true,"new_checkout":false}
├── rate-limits     data: {"api_rpm":1000,"upload_mb":50}
└── maintenance     data: {"enabled":false,"message":""}

Update flow:
  1. Admin updates: setData("/config/feature-flags", newFlags, version)
  2. ZooKeeper replicates via Zab (consistent across ensemble)
  3. All watchers receive NodeDataChanged event
  4. Each service re-reads: getData("/config/feature-flags", watch=true)
  5. Services apply new config immediately

Advantages:
  ✅ Push-based — no polling interval, instant propagation
  ✅ Consistent — all services see the same config (sequential consistency)
  ✅ Versioned — version field prevents lost updates (CAS)
  ✅ Auditable — mzxid and mtime track when config last changed
  ✅ No deployment — change config without restarting services

Atomic multi-config update (multi() operation):
  // Update database AND rate-limits atomically
  zk.multi([
    Op.setData("/config/database", newDbConfig, dbVersion),
    Op.setData("/config/rate-limits", newLimits, limitsVersion),
  ]);
  // Either both succeed or both fail — no partial config state

Double Barrier Pattern:

Phase 1: ENTER (wait for all N processes to arrive)
  1. Each process creates: /barrier/job-123/process-X (ephemeral)
  2. Each process calls: getChildren("/barrier/job-123")
  3. If numChildren < N: set watch, wait
  4. When numChildren == N: barrier is open, all proceed

Phase 2: DO WORK
  All processes perform their computation in parallel

Phase 3: EXIT (wait for all N processes to finish)
  1. Each process deletes its node when done
  2. Each process calls: getChildren("/barrier/job-123")
  3. If numChildren > 0: set watch, wait
  4. When numChildren == 0: all done, barrier complete

Example: MapReduce phase transition
  - 10 mappers must all finish before reducers start
  - Each mapper creates /barrier/map-phase/mapper-X
  - When all 10 exist → map phase complete
  - Reducers were watching → they start
  - Each reducer creates /barrier/reduce-phase/reducer-X
  - When all reducers finish → job complete

Edge case: What if a process crashes mid-barrier?
  - Its ephemeral node is deleted
  - numChildren decreases
  - Barrier may never reach N → need timeout/abort logic

FIFO Queue Pattern:

Queue path: /queue/job-processing

Producer (enqueue):
  zk.create("/queue/job-processing/task-", 
            jobData, PERSISTENT_SEQUENTIAL);
  // Creates: /queue/job-processing/task-0000000001
  //          /queue/job-processing/task-0000000002
  //          /queue/job-processing/task-0000000003

Consumer (dequeue):
  1. children = getChildren("/queue/job-processing")
  2. Sort children by sequence number
  3. Try to process lowest: task-0000000001
  4. Delete task-0000000001 (use version check for atomicity)
  5. If delete fails (another consumer got it) → retry with next

Priority Queue:
  Use different prefixes: /queue/high-001, /queue/med-002, /queue/low-003
  Consumers process high-* first, then med-*, then low-*

⚠️ LIMITATIONS — Why NOT to use ZK as a queue:
  - No consumer groups (every consumer sees all messages)
  - No replay (deleted = gone forever)
  - getChildren returns ALL items (expensive with many items)
  - No backpressure mechanism
  - ZK is not designed for high-throughput data movement
  - Thousands of znodes = memory pressure on ensemble

✅ USE INSTEAD:
  - Kafka: high-throughput, consumer groups, replay, partitioned
  - RabbitMQ: routing, acknowledgments, dead letter queues
  - SQS: managed, scalable, no infrastructure to maintain

Two-Phase Commit via ZooKeeper:

Phase 1: PREPARE (voting)
  Coordinator:
    1. Creates /transactions/txn-42 (persistent)
    2. Creates /transactions/txn-42/participants with list of expected voters
    3. Notifies participants (via watch or external message)

  Each Participant:
    4. Prepares locally (acquire locks, validate)
    5. Votes: creates /transactions/txn-42/votes/participant-X
       data: "COMMIT" or "ABORT"

  Coordinator:
    6. Watches /transactions/txn-42/votes (child watch)
    7. Waits until all expected participants have voted

Phase 2: DECIDE
  If all votes are "COMMIT":
    8. Coordinator sets /transactions/txn-42 data to "COMMIT"
    9. Participants watch this node → see COMMIT → apply changes

  If any vote is "ABORT" (or timeout):
    8. Coordinator sets /transactions/txn-42 data to "ABORT"
    9. Participants watch this node → see ABORT → rollback

Crash recovery:
  - Coordinator crashes before decision: participants see no decision,
    timeout and abort (safe)
  - Coordinator crashes after decision: decision is persisted in ZK,
    participants can read it on recovery
  - Participant crashes before voting: coordinator times out, aborts
  - Participant crashes after voting: recovers, reads decision from ZK