Replication & Consistency

-- SimpleStrategy: NEVER use in production
CREATE KEYSPACE dev_keyspace
WITH replication = {
  'class': 'SimpleStrategy',
  'replication_factor': 3
};

-- NetworkTopologyStrategy: ALWAYS use in production
CREATE KEYSPACE prod_keyspace
WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'dc1': 3,    -- 3 replicas in datacenter 1
  'dc2': 3     -- 3 replicas in datacenter 2
};

-- Why NetworkTopologyStrategy even with 1 DC?
-- Because SimpleStrategy ignores rack placement.
-- NTS ensures replicas are on different racks.
CREATE KEYSPACE single_dc_prod
WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'us-east-1': 3
};

Example: RF=3, Write at QUORUM

Client → Coordinator → Replica 1 (ACK ✓)
                     → Replica 2 (ACK ✓)  ← QUORUM met (2/3), return success
                     → Replica 3 (ACK ✓)  ← arrives later, still written

QUORUM = floor(3/2) + 1 = 2 replicas must acknowledge

Timeline:
  t=0ms   Coordinator sends to all 3 replicas
  t=2ms   Replica 1 acknowledges
  t=3ms   Replica 2 acknowledges → SUCCESS returned to client
  t=15ms  Replica 3 acknowledges (slow but still writes)

If Replica 3 is down:
  - Write still succeeds (2/3 = QUORUM)
  - Hinted handoff stores the write for Replica 3
  - When Replica 3 comes back, hint is replayed

Read at QUORUM (RF=3):

Coordinator sends:
  → Replica 1: full data request
  → Replica 2: digest request (hash of data only)
  
Replica 1 returns: {name: "Alice", timestamp: 1000}
Replica 2 returns: digest(name: "Alice", timestamp: 1000)

If digests match → return data immediately
If digests DON'T match → full read from all replicas
  → Return newest by timestamp
  → Trigger read repair on stale replicas

The "digest" optimization avoids transferring full data
from all replicas when they agree.

Why R + W > RF works (RF=3, W=QUORUM=2, R=QUORUM=2):

Replicas: [A, B, C]

Write "Alice" at QUORUM → written to A and B (2 of 3)
Read at QUORUM → reads from B and C (2 of 3)

Overlap: Node B has the latest write AND is in the read set.
Result: Read returns "Alice" ✓

─────────────────────────────────────────────────────────

Why R + W = RF fails (RF=3, W=ONE=1, R=QUORUM=2):

Write "Alice" at ONE → written to A only (1 of 3)
Read at QUORUM → reads from B and C (2 of 3)

No overlap! Neither B nor C has "Alice" yet.
Result: Read returns stale data ✗

(Eventually B and C get the data via anti-entropy,
 but there's a window of inconsistency)

Setup: RF={'dc1': 3, 'dc2': 3}, Total RF=6

QUORUM (global): floor(6/2) + 1 = 4 replicas across both DCs
  → Must wait for cross-DC network (100-200ms latency)
  → If dc2 is unreachable, writes fail

LOCAL_QUORUM: floor(3/2) + 1 = 2 replicas in local DC only
  → Only local network latency (1-5ms)
  → dc2 being unreachable doesn't affect dc1 operations
  → Data still replicates to dc2 asynchronously

Production recommendation:
  Writes:  LOCAL_QUORUM (fast, survives remote DC failure)
  Reads:   LOCAL_QUORUM (strong consistency within DC)
  
  This gives you:
  - Strong consistency within each DC
  - Eventual consistency across DCs (typically < 100ms)
  - Full availability during DC failures

# Hinted handoff configuration
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000  # 3 hours (default)
# Hints older than this are dropped — node needs full repair

# Hint storage
hints_directory: /var/lib/cassandra/hints
max_hints_delivery_threads: 2
hints_flush_period_in_ms: 10000

Read Repair Example (RF=3, CL=QUORUM):

1. Coordinator reads from Replica A (full) and Replica B (digest)
2. Digests don't match!
3. Coordinator reads full data from all 3 replicas:
   - Replica A: {name: "Alice", ts: 1000}
   - Replica B: {name: "Alice", ts: 1000}
   - Replica C: {name: "Alicia", ts: 800}  ← STALE
4. Coordinator returns newest (ts=1000) to client
5. Coordinator writes {name: "Alice", ts: 1000} to Replica C
6. All replicas now consistent

Read repair is opportunistic — it only fixes data that is read.
Data that is never read remains inconsistent until anti-entropy repair.

# Full repair of a keyspace (repairs all token ranges this node owns)
nodetool repair my_keyspace

# Incremental repair (only repairs data written since last repair)
nodetool repair my_keyspace --incremental

# Repair a specific table
nodetool repair my_keyspace my_table

# Parallel repair (repairs multiple ranges simultaneously)
nodetool repair my_keyspace --parallel

# Sub-range repair (for large clusters, repair in chunks)
nodetool repair my_keyspace -st <start_token> -et <end_token>

# Check repair status
nodetool netstats