Multi-DC & Performance

-- Standard multi-DC keyspace (3 replicas per DC)
CREATE KEYSPACE global_app
WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'us-east-1': 3,
  'eu-west-1': 3,
  'ap-southeast-1': 3
};

-- Asymmetric replication (analytics DC gets fewer replicas)
CREATE KEYSPACE analytics_app
WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'us-east-1': 3,      -- production: full replication
  'us-west-2': 3,      -- DR: full replication
  'analytics-dc': 1    -- analytics: single copy (saves storage)
};

-- Adding a new DC to existing keyspace
ALTER KEYSPACE global_app
WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'us-east-1': 3,
  'eu-west-1': 3,
  'ap-southeast-1': 3,
  'sa-east-1': 3       -- new South America DC
};
-- Then run: nodetool rebuild -- -Dcassandra.rebuild_from=us-east-1
-- on each node in the new DC to stream data

NetworkTopologyStrategy with RF=3 in a DC with 3 racks:

DC: us-east-1 (RF=3)
  Rack A: [Node1, Node2, Node3]
  Rack B: [Node4, Node5, Node6]
  Rack C: [Node7, Node8, Node9]

For partition P (token hashes to Node2):
  Replica 1: Node2 (Rack A) — primary replica
  Replica 2: Node4 (Rack B) — first node in a DIFFERENT rack
  Replica 3: Node7 (Rack C) — first node in yet ANOTHER rack

Result: Partition P survives any single rack failure.
All 3 replicas are in different racks.

⚠️ If you only have 2 racks with RF=3:
  Rack A gets 2 replicas, Rack B gets 1 (or vice versa)
  A single rack failure can lose 2 of 3 replicas!
  Best practice: racks ≥ RF

Why LOCAL_QUORUM is the standard:

Setup: RF={'us-east-1': 3, 'eu-west-1': 3}

With QUORUM (global):
  Required ACKs: floor(6/2) + 1 = 4
  A write in us-east-1 must wait for at least 1 ACK from eu-west-1
  Cross-Atlantic latency: 80-120ms added to EVERY write
  If eu-west-1 is unreachable: ALL writes fail

With LOCAL_QUORUM:
  Required ACKs: floor(3/2) + 1 = 2 (local DC only)
  A write in us-east-1 only waits for 2 local replicas
  Latency: 1-5ms (local network only)
  If eu-west-1 is unreachable: us-east-1 continues normally
  Data replicates to eu-west-1 asynchronously (typically < 100ms)

The trade-off:
  ✅ Strong consistency within each DC
  ✅ Survives remote DC failures
  ✅ Low latency (no cross-DC wait)
  ⚠️ Brief window of inconsistency across DCs (~100ms)
  ⚠️ A write in us-east-1 may not be immediately visible in eu-west-1

Conflict Resolution: Last-Write-Wins (LWW)

Timeline:
  t=0ms    User updates email to "a@new.com" in us-east-1
  t=5ms    Same user updates email to "b@new.com" in eu-west-1
  t=100ms  Replication delivers both writes to both DCs

Resolution (per-cell, not per-row):
  us-east-1 write timestamp: 1705312800000 (t=0ms)
  eu-west-1 write timestamp: 1705312800005 (t=5ms)
  
  Winner: eu-west-1 (higher timestamp) → email = "b@new.com"
  This resolution happens identically on ALL replicas.

Key properties:
  - Resolution is deterministic (same result on every node)
  - Resolution is per-CELL (column), not per-row
  - No data loss detection — the "losing" write is silently discarded
  - Depends on synchronized clocks (NTP is critical!)

Per-cell resolution example:
  us-east-1: UPDATE users SET name='Alice', age=30 WHERE id=1; (ts=100)
  eu-west-1: UPDATE users SET name='Bob', email='b@x.com' WHERE id=1; (ts=105)
  
  Result: name='Bob' (ts=105 wins), age=30 (only write), email='b@x.com' (ts=105 wins)
  Each column resolved independently!

JVM Heap Configuration (jvm.options or jvm11-server.options):

# Heap size: 8 GB is the sweet spot for most workloads
-Xms8G
-Xmx8G
# Never exceed 16 GB — GC pauses become unacceptable
# Rule: heap = min(8GB, 1/4 of RAM, 16GB max)

# Garbage Collector (Cassandra 4.0+):
# ZGC (recommended for Java 17+):
-XX:+UseZGC
-XX:+ZGenerational    # Java 21+

# G1GC (default for Cassandra 4.x on Java 11):
-XX:+UseG1GC
-XX:MaxGCPauseMillis=500
-XX:G1RSetUpdatingPauseTimePercent=5
-XX:InitiatingHeapOccupancyPercent=70

# Off-heap memory (not counted in heap):
# - Bloom filters
# - Compression metadata
# - Key cache (configurable)
# - Chunk cache (Cassandra 4.0+)
# These use native memory — account for them in total RAM planning

# Total RAM budget example (64 GB machine):
#   JVM Heap:     8 GB
#   Off-heap:     4 GB (bloom filters, caches)
#   OS page cache: 48 GB (caches SSTable data files)
#   OS/other:     4 GB

# Key metrics from nodetool:

# Read/write latency per table
nodetool tablestats my_keyspace.my_table | grep -E "latency|SSTable count"

# GC stats
nodetool gcstats

# Thread pool status (look for pending/blocked)
nodetool tpstats | grep -E "Pool|Pending|Blocked"
# Critical pools: ReadStage, MutationStage, CompactionExecutor

# Compaction progress
nodetool compactionstats

# Network streaming (during repair/bootstrap)
nodetool netstats

# JMX metrics for Prometheus/Grafana:
# org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Latency
# org.apache.cassandra.metrics:type=Compaction,name=PendingTasks
# org.apache.cassandra.metrics:type=Storage,name=Load
# org.apache.cassandra.metrics:type=DroppedMessage,scope=MUTATION,name=Dropped

ScyllaDB Shard-Per-Core Architecture:

Traditional Cassandra (thread-per-request):
  ┌─────────────────────────────────────────┐
  │ JVM Heap (shared memory, GC managed)    │
  │                                         │
  │  Thread 1 ──┐                           │
  │  Thread 2 ──┼── Shared data structures  │
  │  Thread 3 ──┤   (locks, contention)     │
  │  Thread N ──┘                           │
  │                                         │
  │  GC pauses affect ALL threads           │
  └─────────────────────────────────────────┘

ScyllaDB (shard-per-core, Seastar framework):
  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
  │ Core 0   │ │ Core 1   │ │ Core 2   │ │ Core N   │
  │          │ │          │ │          │ │          │
  │ Own data │ │ Own data │ │ Own data │ │ Own data │
  │ Own mem  │ │ Own mem  │ │ Own mem  │ │ Own mem  │
  │ Own I/O  │ │ Own I/O  │ │ Own I/O  │ │ Own I/O  │
  │          │ │          │ │          │ │          │
  │ No locks │ │ No locks │ │ No locks │ │ No locks │
  │ No GC    │ │ No GC    │ │ No GC    │ │ No GC    │
  └──────────┘ └──────────┘ └──────────┘ └──────────┘
  
  Each core is independent — no shared state, no locks,
  no garbage collection. Cross-core communication via
  message passing (like Erlang actors).