EnsembleZab ProtocolQuorumLeader ElectionObserversWrite Path

Architecture & The Ensemble

ZooKeeper runs as an ensemble of servers using the Zab atomic broadcast protocol. One leader handles all writes, followers replicate, and quorum ensures consistency even when nodes fail.

40 min read9 sections
01

The Ensemble

A ZooKeeper deployment is called an ensemble — a group of servers (typically 3, 5, or 7) that work together to provide the coordination service. The ensemble must always have an odd number of nodes to ensure a clear majority can be established.

⚖️

The Jury System

An ensemble works like a jury. You need a majority to reach a verdict (quorum). With 5 jurors, you need 3 to agree. If 2 jurors are absent, the remaining 3 can still reach a verdict. But if 3 are absent, the 2 remaining cannot — they must wait. This is why odd numbers matter: with 4 jurors, a 2-2 split is a deadlock. With 5, someone always breaks the tie.

ensemble-sizing.txttext
Ensemble Size vs Fault Tolerance:
═══════════════════════════════════════════════════════════
Nodes │ Quorum │ Can Tolerate │ Notes
═══════════════════════════════════════════════════════════
  1   │   1    │  0 failures  │ Development only
  3   │   2    │  1 failure   │ Minimum production
  5   │   3    │  2 failures  │ Recommended production
  7   │   4    │  3 failures  │ Large/critical deployments
═══════════════════════════════════════════════════════════

Formula: Quorum = ⌊N/2⌋ + 1
         Tolerable failures = N - Quorum = ⌊(N-1)/2⌋

Why NOT even numbers?
  - 4 nodes: quorum = 3, tolerates 1 failure (same as 3 nodes!)
  - 6 nodes: quorum = 4, tolerates 2 failures (same as 5 nodes!)
  - Even numbers add cost without improving fault tolerance
  - They also increase the chance of a tie during leader election

Why 5 is the Sweet Spot

A 3-node ensemble tolerates only 1 failure — during maintenance of one node, you have zero fault tolerance. A 5-node ensemble tolerates 2 failures, meaning you can lose one node to maintenance and still survive an unexpected failure. 7 nodes is rarely needed unless you have extreme availability requirements.

Ensemble Deployment Rules

  • ✅Always use odd numbers (3, 5, 7) — even numbers waste resources
  • ✅Place nodes across failure domains (different racks, AZs, or DCs)
  • ✅All nodes should have similar hardware — the slowest node limits write throughput
  • ✅Use dedicated machines — ZooKeeper is latency-sensitive and GC pauses affect the whole ensemble
  • ✅Network latency between nodes should be low (<10ms) — Zab requires round-trips for every write
1 / 9