The Coordination Problem

Scenario: Two services try to become leader simultaneously

Timeline:
  t=0ms   Service A: "I'll check if there's a leader..."
  t=1ms   Service B: "I'll check if there's a leader..."
  t=2ms   Service A: reads DB → no leader exists
  t=3ms   Service B: reads DB → no leader exists  (race condition!)
  t=4ms   Service A: writes "I am leader" → success
  t=5ms   Service B: writes "I am leader" → success

Result: SPLIT BRAIN — both think they're the leader
  - Service A processes orders 1-50
  - Service B processes orders 1-50 (duplicates!)
  - Data corruption, lost money, angry customers

This is why you need atomic coordination primitives.

// ❌ BROKEN: Naive database-based leader election
async function tryBecomeLeader(nodeId: string) {
  const currentLeader = await db.query(
    'SELECT leader_id FROM leaders WHERE role = $1',
    ['primary']
  );

  // RACE CONDITION: Another node can read the same null value
  // between our read and write!
  if (!currentLeader) {
    await db.query(
      'INSERT INTO leaders (role, leader_id) VALUES ($1, $2)',
      ['primary', nodeId]
    );
    return true; // We think we're leader...
  }

  // PROBLEM: What if the current leader is dead?
  // How long do we wait before declaring it dead?
  // What if it's just slow (GC pause)?
  return false;
}

// ❌ BROKEN: What happens when the leader crashes?
// - The row stays in the database forever
// - No other node can become leader
// - Manual intervention required at 3am

// ❌ BROKEN: What about network partitions?
// - Leader is alive but can't reach the database
// - Other nodes elect a new leader
// - Now TWO leaders exist (split brain)

Scenario: 5-node ZooKeeper ensemble, network partition splits 2|3

Majority partition (3 nodes):
  ✅ Can elect a leader
  ✅ Can serve reads and writes
  ✅ Maintains all guarantees
  → Clients connected here continue normally

Minority partition (2 nodes):
  ❌ Cannot form quorum (need 3 of 5)
  ❌ Cannot serve writes
  ❌ Cannot guarantee reads are current
  → Clients get CONNECTION_LOSS errors
  → Must reconnect to majority partition

Why this is CORRECT behavior:
  - If minority partition served writes, you'd have split brain
  - Two leaders, two versions of truth = data corruption
  - Better to be unavailable than inconsistent for coordination data

ZooKeeper API (simplified):

Create Operations:
  create(path, data, acl, flags)     → creates a znode
    flags: PERSISTENT, EPHEMERAL, SEQUENTIAL, CONTAINER

Read Operations:
  getData(path, watch)               → returns data + stat
  getChildren(path, watch)           → returns child names
  exists(path, watch)                → returns stat or null

Write Operations:
  setData(path, data, version)       → conditional update (CAS)
  delete(path, version)              → conditional delete

Session Operations:
  connect(hosts, timeout)            → establish session
  close()                            → end session cleanly

Multi Operations (3.4+):
  multi(ops[])                       → atomic batch of operations

That's it. ~10 operations. From these, you build:
  - Leader election (ephemeral + sequential)
  - Distributed locks (ephemeral + sequential + watches)
  - Service discovery (ephemeral + getChildren + watches)
  - Configuration (persistent + getData + watches)
  - Barriers (persistent + getChildren + watches)

How Kafka Uses ZooKeeper (pre-KRaft):

/kafka
├── brokers
│   ├── ids
│   │   ├── 0          ← ephemeral: broker 0 is alive
│   │   ├── 1          ← ephemeral: broker 1 is alive
│   │   └── 2          ← ephemeral: broker 2 is alive
│   └── topics
│       └── orders
│           └── partitions
│               ├── 0/state    ← leader: broker 1, ISR: [1,2]
│               └── 1/state    ← leader: broker 2, ISR: [2,0]
├── controller         ← ephemeral: current controller is broker 0
├── config
│   ├── topics/orders  ← topic configuration
│   └── brokers/0      ← broker-level config overrides
└── admin
    └── reassign_partitions  ← partition reassignment state

When broker 1 crashes:
  1. Its ephemeral node /brokers/ids/1 is deleted
  2. Controller (broker 0) gets watch notification
  3. Controller reassigns partition 0 leadership to broker 2
  4. Controller updates /brokers/topics/orders/partitions/0/state
  5. All of this happens in seconds, automatically