Consistency Guarantees

Sequential Consistency Guarantee:

Client A sends: W1 (set /x = 1), W2 (set /y = 2)
Client B sends: W3 (set /x = 3), W4 (set /z = 4)

Valid orderings (respects per-client order):
  ✅ W1, W2, W3, W4  (A's writes first, then B's)
  ✅ W1, W3, W2, W4  (interleaved, but A: W1<W2, B: W3<W4)
  ✅ W3, W1, W4, W2  (interleaved, but A: W1<W2, B: W3<W4)
  ✅ W3, W4, W1, W2  (B's writes first, then A's)

Invalid orderings:
  ❌ W2, W1, W3, W4  (violates A's order: W2 before W1)
  ❌ W1, W4, W3, W2  (violates B's order: W4 before W3)

What this means in practice:
  - If you set config then set "ready=true", all observers
    will see the config BEFORE seeing ready=true
  - Causal ordering within a single client is preserved
  - Cross-client ordering is NOT guaranteed (no global real-time order)

Atomicity Guarantee:

Single operation:
  setData("/config", newData, version)
  → Either the data is fully updated OR the operation fails
  → No partial writes (half the bytes updated)
  → No torn reads (reading during a write)

Multi operation (atomic batch):
  zk.multi([
    Op.create("/txn/step1", data1),
    Op.setData("/txn/step2", data2, version2),
    Op.delete("/txn/step3", version3),
  ]);
  → ALL three operations succeed, OR NONE of them do
  → If step2 fails (wrong version), step1 is rolled back
  → Observers never see an intermediate state

Single System Image:
  Client connected to Server A, reads /config → "v5"
  Client reconnects to Server B (failover)
  Client reads /config → guaranteed to see "v5" or newer
  → Client NEVER goes backward in time after reconnection
  → Implemented via zxid: client sends last-seen zxid,
    server won't serve if it's behind that zxid

The Stale Read Problem:

Timeline:
  t=0  Client A writes /config = "v2" (goes to leader)
  t=1  Leader commits, sends COMMIT to followers
  t=1  Client A gets success response
  t=2  Client B reads /config from Follower 3
       → Follower 3 hasn't received COMMIT yet!
       → Client B reads "v1" (stale!)
  t=3  Follower 3 receives COMMIT, applies "v2"
  t=4  Client B reads /config from Follower 3 → "v2" (now current)

This is CORRECT behavior per ZooKeeper's guarantees!
Sequential consistency doesn't promise real-time reads.

Solutions:

1. sync() — force follower to catch up
   zk.sync("/config");           // blocks until follower is current
   byte[] data = zk.getData("/config", false, stat);  // now guaranteed fresh
   // Cost: one round-trip to leader + wait for sync

2. Read from leader (not standard API, but some clients support it)
   // Always gets latest committed value
   // Cost: all reads go to one server (no read scaling)

3. Use watches instead of polling
   // Watch fires AFTER the update is committed
   // So re-reading after a watch event always gets the new value

When stale reads are OK:
  - Reading config that changes rarely
  - Service discovery (slightly stale list is fine)
  - Monitoring/dashboards

When stale reads are NOT OK:
  - Checking if you still hold a lock (use session events instead)
  - Reading a value you just wrote from a different server

zxid Structure (64 bits):

┌─────────────────────────────────────────────────────────┐
│  High 32 bits: EPOCH  │  Low 32 bits: COUNTER           │
│  (leader term)        │  (transaction within epoch)      │
└─────────────────────────────────────────────────────────┘

Example: zxid = 0x0000000300000047
  Epoch:   3  (third leader since cluster started)
  Counter: 71 (71st transaction in this epoch)

Properties:
  - Globally unique (no two transactions share a zxid)
  - Monotonically increasing (zxid N+1 is always after zxid N)
  - Encodes causality (higher zxid = happened later in ZK's view)
  - Epoch increases on every leader election
  - Counter resets to 0 on new epoch

Where zxid appears:
  - Stat.czxid  → zxid that created this znode
  - Stat.mzxid  → zxid that last modified this znode
  - Stat.pzxid  → zxid of last child modification
  - Watch events → include the zxid that triggered the watch
  - Client session → tracks last-seen zxid for ordering

Uses:
  - Fencing tokens (lock holder's czxid)
  - Ordering events (which happened first?)
  - Leader election (highest zxid = most up-to-date)
  - Follower sync (send all transactions after zxid X)

Epoch Progression:

Epoch 1 (Leader: Server A)
  zxid 0x0000000100000001  → first transaction
  zxid 0x0000000100000002  → second transaction
  ...
  zxid 0x000000010000002F  → 47th transaction
  [Server A crashes]

Epoch 2 (Leader: Server B)
  zxid 0x0000000200000001  → first transaction in new epoch
  zxid 0x0000000200000002  → second transaction
  ...
  [Server B steps down gracefully]

Epoch 3 (Leader: Server C)
  zxid 0x0000000300000001  → first transaction in epoch 3

Why epochs prevent zombie leaders:
  1. Server A was leader in epoch 1
  2. Server A gets network-partitioned (but doesn't know it's not leader)
  3. Server B elected as leader in epoch 2
  4. Server A recovers, tries to send proposals with epoch 1
  5. Followers reject: "I'm in epoch 2, your epoch 1 is stale"
  6. Server A realizes it's no longer leader, becomes follower

Without epochs:
  - Old leader could commit transactions after new leader starts
  - Two leaders writing simultaneously = inconsistency
  - Epochs are the "term number" that prevents this

Performance Bottlenecks and Solutions:

1. Write Latency (biggest factor: disk fsync)
   Problem: Every write fsyncs to transaction log before ACK
   Solution: Use SSDs (1-2ms fsync vs 10-15ms HDD)
   Solution: Separate dataLogDir from dataDir (dedicated disk for txn log)

2. Write Throughput (bounded by leader)
   Problem: Single leader processes all writes sequentially
   Solution: Batch writes where possible (multi() operation)
   Solution: Reduce write frequency (batch config updates)
   Anti-pattern: Don't write metrics/logs to ZK

3. Read Latency (usually not a problem)
   Problem: Stale reads from followers
   Solution: Use sync() only when freshness is critical
   Solution: Connect to leader for critical reads (rare)

4. Large Ensemble Overhead
   Problem: More followers = more proposals to send
   Solution: Use observers for read scaling (don't affect quorum)
   Solution: Keep voting ensemble small (5-7 nodes max)

5. Watch Storm
   Problem: Many clients watching same node = many notifications
   Solution: Use hierarchical watches (watch parent, not all children)
   Solution: Batch watch processing in client

Typical production numbers (5-node ensemble, SSDs):
  Read throughput:  ~150,000 ops/sec (distributed across 5 nodes)
  Write throughput: ~30,000 ops/sec (all through leader)
  Read latency:    ~0.5ms (p99: ~2ms)
  Write latency:   ~5ms (p99: ~15ms)
  Ensemble size:   ~100K-500K znodes comfortably