Sessions & Watches

Session Lifecycle:

1. CONNECTING
   Client initiates TCP connection to a ZK server
   Sends: ConnectRequest(sessionTimeout, sessionId=0, password=empty)
   Server responds: ConnectResponse(sessionId, negotiatedTimeout, password)

2. CONNECTED
   Client can now perform operations
   Client sends periodic heartbeats (PING)
   Server tracks session liveness

3. DISCONNECTED (temporary)
   TCP connection lost (network blip, server restart)
   Client enters CONNECTING state
   Client tries other servers in the ensemble
   Session is still alive on the server side (timeout hasn't expired)

4. RECONNECTED
   Client connects to another server
   Sends: ConnectRequest(sessionTimeout, sessionId=X, password=Y)
   Server validates session is still alive → CONNECTED again
   All ephemeral nodes and watches are preserved!

5. EXPIRED (terminal)
   Session timeout elapsed without heartbeat
   Server deletes all ephemeral nodes owned by this session
   All watches are removed
   Client receives SESSION_EXPIRED event
   Client must create a completely new session

Session state machine:
  CONNECTING → CONNECTED → DISCONNECTED → CONNECTING → CONNECTED
                                                     → EXPIRED (terminal)

Timeout Negotiation:

Server config (zoo.cfg):
  tickTime = 2000        # Base time unit in ms
  minSessionTimeout = 4000   # 2 × tickTime (minimum)
  maxSessionTimeout = 40000  # 20 × tickTime (maximum)

Client requests: sessionTimeout = 30000 (30 seconds)
Server responds: negotiatedTimeout = 30000 (within bounds ✅)

Client requests: sessionTimeout = 1000 (1 second)
Server responds: negotiatedTimeout = 4000 (clamped to minimum)

Heartbeat interval:
  Client sends PING every: negotiatedTimeout / 3
  Example: 30s timeout → PING every 10 seconds

  Why timeout/3?
  - Gives 2 missed heartbeats before expiry
  - Accounts for network latency and GC pauses
  - Client: "I'll ping at 10s, 20s" → Server expires at 30s

Server-side tracking:
  - Server checks session liveness every tickTime (2s)
  - If no heartbeat received within sessionTimeout → expire
  - The leader is responsible for session expiry decisions

// Handling session expiry (conceptual)
zk.addAuthInfo("digest", "user:pass".getBytes());

// Register a session watcher
Watcher sessionWatcher = (event) -> {
  if (event.getState() == KeeperState.Expired) {
    // SESSION EXPIRED — everything is gone!
    // - All our ephemeral nodes: DELETED
    // - All our watches: REMOVED  
    // - All pending operations: CANCELLED
    
    // We MUST:
    // 1. Create a completely new ZooKeeper client
    // 2. Re-create all ephemeral nodes (re-register as service, re-acquire locks)
    // 3. Re-set all watches
    // 4. Re-initialize application state
    
    // We CANNOT:
    // - Reconnect with the old session (it's gone forever)
    // - Assume our ephemeral nodes still exist
    // - Assume we still hold any locks
    
    reinitialize();
  } else if (event.getState() == KeeperState.Disconnected) {
    // DISCONNECTED — session might still be alive!
    // Don't panic yet. The client library will try to reconnect.
    // If it reconnects before timeout, everything is preserved.
    log.warn("Disconnected from ZK — attempting reconnect...");
  }
};

Reconnection Scenarios:

Scenario 1: Quick reconnect (SUCCESS)
  t=0s    Client connected to Server A
  t=1s    Server A crashes
  t=1s    Client enters DISCONNECTED state
  t=2s    Client tries Server B → connection established
  t=2s    Client sends session ID + password to Server B
  t=2s    Server B validates session is still alive → CONNECTED
  Result: ✅ Session preserved, ephemeral nodes intact, watches intact

Scenario 2: Slow reconnect (SUCCESS, barely)
  t=0s    Client connected to Server A
  t=1s    Network partition begins
  t=1s    Client enters DISCONNECTED state
  t=5s    Client tries Server B → fails (also partitioned)
  t=10s   Client tries Server C → fails
  t=25s   Network heals, Client connects to Server B
  t=25s   Session timeout is 30s → session still alive!
  Result: ✅ Session preserved (reconnected before 30s timeout)

Scenario 3: Too slow (EXPIRED)
  t=0s    Client connected to Server A
  t=1s    Network partition begins
  t=30s   Server-side: session timeout reached → SESSION EXPIRED
  t=30s   Ephemeral nodes deleted, watches removed
  t=45s   Network heals, Client connects to Server B
  t=45s   Client sends old session ID → Server says "EXPIRED"
  Result: ❌ Session gone, must create new session

Key insight: The client doesn't know if its session is expired
until it successfully reconnects. During DISCONNECTED state,
it should assume the session MIGHT still be alive and keep trying.

Watch Registration and Triggering:

DATA WATCHES (registered by getData or exists):
  // Register: "notify me when /config/db data changes"
  byte[] data = zk.getData("/config/db", true, stat);
  
  // Triggered by:
  //   NodeDataChanged  → someone called setData on /config/db
  //   NodeDeleted      → someone deleted /config/db
  //   NodeCreated      → (only for exists watch on non-existent node)

CHILD WATCHES (registered by getChildren):
  // Register: "notify me when children of /services change"
  List<String> children = zk.getChildren("/services", true);
  
  // Triggered by:
  //   NodeChildrenChanged → child added or removed under /services
  //   NodeDeleted         → /services itself was deleted
  
  // NOT triggered by:
  //   ❌ Data changes in children (only structure changes)
  //   ❌ Data changes in /services itself

IMPORTANT: Watches are one-time triggers!
  After firing once, the watch is gone.
  You must re-register it to get the next notification.
  
  // Pattern: read + watch in a loop
  while (true) {
    data = zk.getData("/config/db", watchCallback, stat);
    // ... use data ...
    // watchCallback fires → loop re-reads with new watch
  }

Watch Ordering Guarantee:

Client A sets watch on /config
Client B updates /config to "v2"
Client B updates /config to "v3"

Client A receives:
  1. Watch event: NodeDataChanged on /config
  2. (Client A re-reads /config → gets "v3", NOT "v2")

Key insight: Client A might "miss" seeing "v2" — but that's OK!
The watch told it "something changed." The re-read gets the LATEST value.
For coordination, you care about the current state, not the history.

No-Miss Guarantee (with proper re-registration):
  If you always re-register your watch immediately after it fires
  (in the same callback), you will never miss a change. The pattern:

  void watchCallback(WatchedEvent event) {
    // Watch fired — re-read with new watch immediately
    byte[] newData = zk.getData("/config", this, stat);
    // Process newData...
  }

  Between the watch firing and re-registration, changes are
  captured by the re-read (you get the latest state).

// Classic one-time watch pattern (pre-3.6)
// Problem: gap between watch fire and re-registration

void watchConfig() {
  byte[] data = zk.getData("/config", event -> {
    // Watch fired! But between now and re-registration,
    // another change could happen that we'd miss...
    // (In practice, the re-read catches it, but the pattern is complex)
    watchConfig(); // re-register
  }, stat);
  processConfig(data);
}

// Persistent watch (3.6+) — no re-registration needed
zk.addWatch("/config", event -> {
  // This fires for EVERY change, no re-registration needed
  byte[] newData = zk.getData("/config", null, stat);
  processConfig(newData);
}, AddWatchMode.PERSISTENT);

// Persistent recursive watch — watches entire subtree
zk.addWatch("/services", event -> {
  // Fires for any change under /services (any depth)
  // NodeCreated, NodeDeleted, NodeDataChanged for any descendant
  refreshServiceRegistry();
}, AddWatchMode.PERSISTENT_RECURSIVE);

// Remove when done
zk.removeWatches("/config", watcher, WatcherType.Any, false);