ZooKeeper vs Alternatives

etcd Key Differences:

1. Flat namespace (not hierarchical)
   ZooKeeper: /services/payment/instance-1 (tree structure)
   etcd: key="/services/payment/instance-1" (flat, but prefix queries work)
   
   etcd range query: get all keys with prefix "/services/payment/"
   → Functionally similar to getChildren, but flat

2. Streaming watches (not one-time)
   ZooKeeper: watch fires once, must re-register
   etcd: watch is a stream — keeps sending events until cancelled
   
   etcd also provides revision-based watches:
   "Watch key X starting from revision 42"
   → Never miss events, even if client was disconnected

3. Leases instead of sessions
   ZooKeeper: ephemeral nodes tied to client session
   etcd: leases with explicit TTL, attached to one or more keys
   
   // Create a lease (like a session)
   lease = client.lease.grant(ttl=30)
   // Attach keys to the lease
   client.put("/services/payment/inst-1", "10.0.1.5", lease=lease)
   // Keep alive (like heartbeat)
   client.lease.keepAlive(lease)
   // If keepAlive stops → lease expires → keys deleted

4. Mini-transactions (more powerful than multi)
   // Atomic compare-and-swap with conditions
   client.txn()
     .if(key("/leader").value == "")
     .then(put("/leader", myId, lease))
     .else(get("/leader"))
     .commit()

// Curator Leader Election (vs raw ZooKeeper)

// RAW ZooKeeper (50+ lines, error-prone):
//   - Create ephemeral sequential node
//   - getChildren, sort, check if lowest
//   - Watch previous node
//   - Handle watch re-registration
//   - Handle session expiry
//   - Handle edge cases (node deleted before watch set)

// CURATOR (5 lines, battle-tested):
LeaderLatch latch = new LeaderLatch(client, "/election", myId);
latch.addListener(new LeaderLatchListener() {
  public void isLeader() { startLeaderDuties(); }
  public void notLeader() { stopLeaderDuties(); }
});
latch.start();

// Curator Distributed Lock:
InterProcessMutex lock = new InterProcessMutex(client, "/locks/resource");
lock.acquire(30, TimeUnit.SECONDS);  // blocks until acquired or timeout
try {
  // critical section
} finally {
  lock.release();
}

// Curator Service Discovery:
ServiceDiscovery<InstanceDetails> discovery = ServiceDiscoveryBuilder
  .builder(InstanceDetails.class)
  .client(client)
  .basePath("/services")
  .thisInstance(myInstance)
  .build();
discovery.start();  // registers this instance
// Other services query: discovery.queryForInstances("payment")

Consul's Advantages Over ZooKeeper:

1. Built-in Health Checking
   ZooKeeper: session timeout is the only liveness signal
     → Service is "alive" if session exists (even if it's broken)
   
   Consul: multiple health check types
     → HTTP: GET /health returns 200?
     → TCP: can connect to port 8080?
     → Script: custom check script returns 0?
     → gRPC: health check RPC succeeds?
     → TTL: service must heartbeat within interval
   
   Consul removes unhealthy services from discovery BEFORE they crash.

2. DNS Interface
   ZooKeeper: clients need ZK client library
   Consul: services discoverable via DNS
     → dig payment.service.consul
     → Returns healthy instances only
     → Works with any language, no special library needed

3. Multi-Datacenter Native
   ZooKeeper: single ensemble, observers in remote DC
   Consul: separate server clusters per DC, WAN gossip between DCs
     → Each DC has its own Raft cluster (fast local writes)
     → Cross-DC queries via WAN federation
     → Prepared queries can failover across DCs

4. Service Mesh (Connect)
   ZooKeeper: coordination only
   Consul: also provides service-to-service mTLS, traffic routing,
           intentions (authorization), and sidecar proxy management

When to Mention ZooKeeper in System Design:

1. Leader Election
   "We need exactly one instance processing payments at a time.
    We'll use ZooKeeper for leader election — ephemeral nodes
    ensure automatic failover if the leader crashes."

2. Service Discovery
   "Services register with ZooKeeper using ephemeral nodes.
    Consumers watch the service path and get notified instantly
    when instances join or leave."

3. Distributed Configuration
   "Feature flags are stored in ZooKeeper. All services watch
    the config node and apply changes instantly — no deployment
    needed, no polling delay."

4. Distributed Locking
   "To prevent duplicate payment processing, we use a ZooKeeper
    distributed lock. Ephemeral nodes ensure the lock is released
    if the holder crashes."

5. Cluster Membership
   "Each node registers an ephemeral node in ZooKeeper. The
    controller watches the membership path to detect failures
    and trigger rebalancing."

How to Describe It Correctly:
  ✅ "A distributed coordination service that provides leader
      election, locks, and service discovery primitives"
  ✅ "A CP system — chooses consistency over availability"
  ✅ "Uses consensus (Zab) to replicate state across an ensemble"
  
  ❌ "A distributed database" (it's not — 1MB limit, not for app data)
  ❌ "A message queue" (it's not — use Kafka)
  ❌ "Always available" (it's CP — unavailable without quorum)

Level of Detail:
  - Mention it exists and what it does: always
  - Explain the mechanism (ephemeral nodes, watches): if asked
  - Discuss internals (Zab, quorum, zxid): only if specifically asked