Design a Messaging App (WhatsApp / Telegram)

Given:
  100B messages/day
  500M DAU (daily active users)
  Average user sends ~40 messages/day and receives ~160 (groups amplify)

Message throughput:
  100B / 86,400s ≈ 1.16M messages/sec (average)
  Peak ≈ 3× average ≈ 3.5M messages/sec (evening hours, global)

Per-user perspective:
  Average: 200 messages/day (sent + received) = 1 message every 7 minutes
  Power user: 2000 messages/day = 1 message every 43 seconds

Implication:
  → This is NOT a high-QPS-per-server problem like URL shortener.
  → The challenge is CONCURRENT CONNECTIONS, not request throughput.
  → Each WebSocket server handles ~500K connections but only ~50K msg/sec.

Concurrent connections (peak):
  500M DAU × ~10% online at any moment ≈ 50M concurrent connections
  (WhatsApp reported ~50M concurrent in 2020 with 2B users)

Per-server capacity:
  Optimized WebSocket server (Go/Rust/C++): ~500K connections per box
  With 64GB RAM: ~20KB per connection × 500K = 10GB for connections
  Remaining RAM for message routing, buffers, TLS state

Server fleet:
  50M connections ÷ 500K per server = 100 WebSocket servers (minimum)
  With 2× headroom for failover: ~200 WebSocket servers globally

Implication:
  → Connection state is the bottleneck, not CPU.
  → Server failure means 500K users instantly disconnect and must reconnect.
  → Need connection draining and graceful handoff during deploys.

Per-message estimate:
  message_id       : 16 bytes (UUID or Snowflake)
  conversation_id  : 16 bytes
  sender_id        : 16 bytes
  content          : 200 bytes (average text message)
  timestamp        : 8 bytes
  status           : 1 byte
  metadata         : 50 bytes (reply_to, forwarded, etc.)
  ─────────────────────────────
  Raw message      ≈ 300 bytes
  With Cassandra overhead (tombstones, bloom filter, index) ≈ 500 bytes

Daily storage growth (text only):
  100B messages/day × 500 bytes ≈ 50 TB/day
  → ~18 PB/year (text messages alone)

Media storage:
  Assume 5% of messages have media, average 2MB per media file
  100B × 5% × 2MB = 10 PB/day (!)
  → Media dominates storage by 200×. Object storage (S3) is mandatory.

Implication:
  → Text messages: Cassandra/ScyllaDB with TTL-based compaction
  → Media: S3 with lifecycle policies (hot → warm → cold)
  → Message retention: 30 days on server (WhatsApp model) vs forever (Telegram)
  → With 30-day retention: ~1.5 PB hot text storage at any time

Text messages:
  Inbound:  1.16M msg/sec × 300 bytes ≈ 350 MB/s ≈ 2.8 Gbps
  Outbound: Same (every message sent is delivered to 1+ recipients)
  With group amplification (avg 3 recipients): ~8.4 Gbps outbound

Media:
  5M media messages/sec × 2MB average = 10 TB/sec (!)
  → This is why media goes through CDN, not through chat servers.
  → Chat servers only relay a URL/thumbnail, not the actual file.

WebSocket overhead:
  50M connections × 64-byte heartbeat every 30s = 100 MB/s just for keepalives

Implication:
  → Text bandwidth is manageable on modern NICs (25-100 Gbps per server)
  → Media MUST be decoupled — upload to S3, share URL via message
  → Heartbeat traffic is non-trivial at scale — batch or reduce frequency

Online/offline status updates:
  500M DAU × 2 transitions/day (online + offline) = 1B events/day
  Peak: ~30K status changes/sec

Typing indicators:
  Assume 10% of DAU typing at any moment, 1 event/sec while typing
  50M × 1/sec = 50M typing events/sec (!)
  → Typing indicators are EPHEMERAL — never persisted, never queued.
  → Delivered best-effort via the existing WebSocket connection.
  → If recipient is offline, typing indicator is simply dropped.

Delivery receipts:
  100B messages × 2 receipts each (delivered + read) = 200B receipts/day
  ≈ 2.3M receipts/sec
  → Receipts are batched and sent periodically, not per-message.

Implication:
  → Presence is a hot-path problem but NOT a storage problem.
  → Typing indicators must NEVER hit a database or queue.
  → Delivery receipts can be batched (every 5s) to reduce write amplification.

{
  "type": "message | ack | typing | presence | receipt | sync",
  "id": "msg_7f3a9b2c",
  "timestamp": 1716300000000,
  "payload": { ... }
}

// Client → Server: Send a message
{
  "type": "message",
  "id": "msg_7f3a9b2c",
  "timestamp": 1716300000000,
  "payload": {
    "conversation_id": "conv_abc123",
    "content": {
      "type": "text",
      "body": "Hey, are you free for lunch?"
    },
    "reply_to": null,
    "mentions": []
  }
}

// Server → Client: Acknowledgment (message accepted)
{
  "type": "ack",
  "id": "msg_7f3a9b2c",
  "timestamp": 1716300000050,
  "payload": {
    "status": "accepted",
    "sequence": 48291,
    "server_timestamp": 1716300000045
  }
}

// Server → Client: Rejection (validation failed)
{
  "type": "ack",
  "id": "msg_7f3a9b2c",
  "timestamp": 1716300000050,
  "payload": {
    "status": "rejected",
    "reason": "CONTENT_TOO_LONG",
    "max_length": 65536
  }
}

// Recipient → Server: Message delivered to device
{
  "type": "receipt",
  "id": "rcpt_x9k2m",
  "payload": {
    "message_ids": ["msg_7f3a9b2c", "msg_8a4b1d3e"],
    "conversation_id": "conv_abc123",
    "status": "delivered",
    "device_id": "dev_iphone_14"
  }
}

// Recipient → Server: Messages read by user
{
  "type": "receipt",
  "id": "rcpt_p3n7q",
  "payload": {
    "conversation_id": "conv_abc123",
    "status": "read",
    "read_up_to_sequence": 48291
  }
}

// Server → Sender: Forward receipt
{
  "type": "receipt",
  "id": "rcpt_fwd_1",
  "payload": {
    "conversation_id": "conv_abc123",
    "recipient_id": "user_bob",
    "status": "delivered",
    "message_ids": ["msg_7f3a9b2c"]
  }
}

// Client → Server: User is typing
{
  "type": "typing",
  "id": "typ_1a2b3c",
  "payload": {
    "conversation_id": "conv_abc123",
    "action": "started"
  }
}

// Automatically expires after 5 seconds if no "started" renewal.
// No "stopped" event needed — absence of "started" = stopped.

GET /api/v1/conversations/conv_abc123/messages?before_sequence=48291&limit=50 HTTP/1.1
Authorization: Bearer <jwt>

--- 200 OK ---
{
  "messages": [
    {
      "id": "msg_6e2a8b1d",
      "sequence": 48241,
      "sender_id": "user_alice",
      "content": { "type": "text", "body": "See you at noon!" },
      "timestamp": 1716299500000,
      "status": "read"
    },
    ...
  ],
  "cursor": {
    "before_sequence": 48241,
    "has_more": true
  }
}

1. CONNECT
   Client opens WebSocket: wss://chat.app/ws
   Server assigns connection to a WebSocket server via load balancer
   (sticky by user_id hash for connection affinity)

2. AUTHENTICATE
   First frame must be auth: { "type": "auth", "token": "<jwt>" }
   Server validates JWT, extracts user_id
   Registers (user_id, device_id) → (server_id, connection_id) in Redis

3. SYNC
   Client sends last-known sequence per conversation
   Server pushes any missed messages since that sequence
   This handles the "offline gap" — messages received while disconnected

4. ACTIVE
   Bidirectional message flow
   Server sends heartbeat every 30s; client responds with pong
   If 3 heartbeats missed → server closes connection, marks user offline

5. DISCONNECT
   Client sends close frame (graceful) or TCP drops (ungraceful)
   Server removes routing entry from Redis
   Queues any in-flight messages to offline inbox for push notification

CREATE TABLE messages (
  conversation_id  UUID,           -- partition key
  sequence         BIGINT,         -- clustering column (monotonic per conversation)
  message_id       UUID,           -- globally unique, client-generated (idempotency)
  sender_id        UUID,
  content_type     TEXT,           -- 'text', 'image', 'video', 'audio', 'document'
  content_body     TEXT,           -- encrypted text or media URL
  content_metadata FROZEN<MAP<TEXT, TEXT>>,  -- thumbnail_url, file_size, duration, etc.
  reply_to_seq     BIGINT,         -- NULL if not a reply
  status           TEXT,           -- 'sent', 'delivered', 'read' (per-conversation aggregate)
  created_at       TIMESTAMP,
  deleted_at       TIMESTAMP,      -- soft delete (message deleted for everyone)
  PRIMARY KEY ((conversation_id), sequence)
) WITH CLUSTERING ORDER BY (sequence DESC)
  AND default_time_to_live = 2592000;  -- 30-day TTL (WhatsApp model)

-- Why DESC: "fetch latest 50 messages" is the hot query.
-- Cassandra reads clustering columns in order — DESC means latest first.

CREATE TABLE conversations (
  conversation_id    UUID,
  type               TEXT,           -- '1:1' or 'group'
  name               TEXT,           -- NULL for 1:1, group name for groups
  avatar_url         TEXT,
  created_by         UUID,
  created_at         TIMESTAMP,
  last_message_seq   BIGINT,         -- latest sequence (for sync)
  last_message_preview TEXT,         -- "Alice: Hey, are you free?" (denormalized)
  last_message_at    TIMESTAMP,      -- for inbox sorting
  PRIMARY KEY ((conversation_id))
);

-- User's inbox: which conversations does user X belong to?
CREATE TABLE user_conversations (
  user_id            UUID,           -- partition key
  last_activity_at   TIMESTAMP,      -- clustering (for inbox sort)
  conversation_id    UUID,
  unread_count       INT,
  muted              BOOLEAN,
  pinned             BOOLEAN,
  PRIMARY KEY ((user_id), last_activity_at, conversation_id)
) WITH CLUSTERING ORDER BY (last_activity_at DESC);

-- Why denormalize last_message_preview?
-- The inbox query needs it. Joining messages table for every conversation
-- in the inbox would be N+1 queries. Denormalize on write, read once.

-- "Who is in this conversation?" (needed for message fan-out)
CREATE TABLE conversation_members (
  conversation_id  UUID,           -- partition key
  user_id          UUID,           -- clustering column
  role             TEXT,           -- 'admin', 'member'
  joined_at        TIMESTAMP,
  last_read_seq    BIGINT,         -- per-user read cursor (for unread count)
  PRIMARY KEY ((conversation_id), user_id)
);

-- For a 1:1 conversation: exactly 2 members
-- For a group: up to 100K members (Telegram-scale)
-- For groups > 1000 members, this partition gets large.
-- Mitigation: bucket by (conversation_id, bucket_id) where bucket = user_id % 10

Key pattern: user:{user_id}:connections
Type: Hash (supports multi-device)

HSET user:alice:connections
  "dev_iphone_14"  "ws-server-042:conn_8a3b"
  "dev_macbook"    "ws-server-117:conn_2f9c"

TTL: 60 seconds (refreshed by heartbeat)
If TTL expires → user is offline → route to push notification

Additional keys:
  user:{user_id}:presence → "online" | "offline" | timestamp (last_seen)
  user:{user_id}:typing:{conversation_id} → TTL 5s (auto-expires)

Why Redis?
  → Sub-millisecond lookups for routing decisions
  → TTL handles cleanup automatically (no stale connections)
  → Pub/Sub for cross-server message routing

Option A: Redis SET with TTL
  SETEX dedup:{message_id} 86400 "1"
  → Check before processing: if EXISTS → already processed, return cached ack
  → 24-hour TTL covers all reasonable retry windows

Option B: Cassandra table (for durability)
  CREATE TABLE message_dedup (
    message_id UUID PRIMARY KEY,
    processed_at TIMESTAMP
  ) WITH default_time_to_live = 86400;

Choice: Redis for speed (sub-ms check on every message).
Cassandra as fallback if Redis loses the key (rare, acceptable).

Total budget: 500ms end-to-end (sender tap → recipient screen)

Sender side:
  0ms    ─  User taps send
  5ms    ─  Client encrypts (E2E), constructs frame
  10ms   ─  WebSocket frame sent over network

Server side:
  15ms   ─  WS Gateway receives frame, validates auth
  20ms   ─  Chat Service: dedup check (Redis EXISTS) → 0.5ms
  25ms   ─  Chat Service: assign sequence (Redis INCR) → 0.5ms
  30ms   ─  Chat Service: persist to Cassandra (async, non-blocking)
  35ms   ─  Chat Service: lookup recipient connection (Redis HGET) → 0.5ms
  40ms   ─  Chat Service: forward to recipient's WS server (internal RPC)

Recipient side:
  45ms   ─  WS Gateway B pushes frame to recipient's connection
  50ms   ─  Network transit to recipient device
  55ms   ─  Client decrypts, renders message
  60ms   ─  Client sends delivery receipt back

Total on happy path: ~60ms (well under 500ms budget)
─────────────────────────────────────────────────────
Worst case (cross-region, Cassandra write blocks): ~200ms
Still under budget with 2.5× headroom.

On WebSocket connect:
  HSET user:{user_id}:connections {device_id} "{server_id}:{conn_id}"
  EXPIRE user:{user_id}:connections 60   // refreshed by heartbeat

On message delivery:
  connections = HGETALL user:{recipient_id}:connections
  // Returns: { "iphone": "ws-042:conn_8a3b", "macbook": "ws-117:conn_2f9c" }
  // → Forward message to BOTH servers (multi-device)

On disconnect:
  HDEL user:{user_id}:connections {device_id}
  // If hash is now empty → user is fully offline

Heartbeat (every 30s):
  EXPIRE user:{user_id}:connections 60
  // If client misses 2 heartbeats (60s), key auto-expires → offline

1. DETECTION (5-10 seconds)
   Health checker (separate service) pings each WS server every 5s.
   3 missed pings → declare server dead.
   Alternative: server registers with TTL in Redis; expiry = dead.

2. CLEANUP (immediate)
   Bulk-delete all connection entries for the dead server:
   SCAN for keys where value contains "ws-server-042"
   → Mark all those users as "offline" in presence
   → Any in-flight messages for those users → route to offline inbox

3. RECONNECTION (client-driven, staggered)
   Clients detect disconnect (WebSocket onclose event).
   Each client waits: random(0, 5000ms) + exponential_backoff
   → Prevents thundering herd (500K clients reconnecting simultaneously)
   → Load balancer distributes across remaining healthy servers

4. SYNC (on reconnect)
   Client sends last-known sequence per conversation.
   Sync Service pushes missed messages.
   → No messages lost — they're in Cassandra, just not yet delivered.

5. REBALANCING (background)
   After reconnection storm settles, connections are unevenly distributed.
   Gradual rebalancing: periodically ask 5% of connections on overloaded
   servers to reconnect (server sends "please reconnect" frame).

Server → Client: PING (every 30 seconds)
Client → Server: PONG (must arrive within 5 seconds)

If PONG not received:
  1. Server marks connection as "suspect"
  2. Server sends one more PING (retry)
  3. If still no PONG after 5s → connection is dead
  4. Server closes WebSocket, removes from Redis registry
  5. Any pending messages → route to offline inbox
  6. Client detects close → reconnects with jittered backoff → syncs

Overhead calculation:
  50M connections × 1 ping/30s = 1.67M ping-pong exchanges/sec
  Each ping/pong is ~2 bytes payload → ~3.3 MB/s total
  → Trivial overhead. Well worth the guaranteed detection time.

Why not just use WebSocket protocol-level ping/pong?
  → Browser WebSocket API doesn't expose protocol-level pings to JS
  → Application-level heartbeats give us control over timing and payload
  → We can piggyback useful data on heartbeats (sequence numbers, presence)

1. Mark server as "draining" in service registry
   → Load balancer stops sending NEW connections to this server
   → Existing connections continue working

2. Send "reconnect" frame to clients in batches
   → 10% of connections every 30 seconds
   → Each client reconnects to a different (healthy) server
   → Total drain time: ~5 minutes for 500K connections

3. Wait for all connections to close (or timeout after 10 min)
   → Remaining stubborn connections get force-closed

4. Shut down server, deploy new version, restart
   → Server rejoins the pool, load balancer sends new connections

Why not just restart?
  → 500K simultaneous disconnects = thundering herd
  → Gradual drain = smooth redistribution across the fleet
  → Users experience a brief reconnect (~1s), not a service outage

Target: 500K concurrent connections per server

Memory breakdown:
  TCP buffers (kernel)     : 500K × 8KB  = 4 GB
  TLS state               : 500K × 5KB  = 2.5 GB
  Application state       : 500K × 4KB  = 2 GB   (user_id, device, auth, buffers)
  Message routing cache   : 1 GB                  (hot user→server mappings)
  Go/Rust runtime         : 2 GB                  (GC overhead, goroutine stacks)
  OS + misc               : 2.5 GB
  ─────────────────────────────────────────────────
  Total                   : ~14 GB (of 64 GB available)
  Headroom                : 50 GB for burst buffers and message queuing

CPU:
  Heartbeat processing    : 500K / 30s = 16.7K heartbeats/sec → trivial
  Message routing         : ~50K messages/sec per server → ~2 cores
  TLS encrypt/decrypt     : ~3 cores (AES-NI hardware acceleration)
  Frame parsing           : ~1 core
  ─────────────────────────────────────────────────
  Total                   : ~7 cores active (of 32 available)
  Bottleneck              : File descriptors (ulimit -n 1048576) and memory, not CPU

Per-conversation sequence counter (Redis):
  Key: seq:{conversation_id}
  Operation: INCR seq:{conversation_id} → returns next sequence

Example flow:
  Alice sends "Hey" to conv_abc → sequence 101
  Bob sends "Hi!" to conv_abc → sequence 102
  Alice sends "Lunch?" to conv_abc → sequence 103

  Even if Alice's "Lunch?" arrives at the server before Bob's "Hi!"
  (due to network jitter), the server assigns sequences in arrival order.
  The server is the single source of truth for ordering within a conversation.

Why not timestamps?
  Alice's phone clock: 12:00:01.000 (correct)
  Bob's phone clock:   12:00:00.500 (30 seconds behind)
  → Bob's message would appear BEFORE Alice's even though Alice sent first.
  → Server-assigned sequence eliminates client clock dependency entirely.

SENDER SIDE:
  1. Client sends message with client-generated UUID (msg_7f3a9b2c)
  2. Client starts retry timer (5 seconds)
  3. If no server ACK within 5s → resend same frame (same msg_id)
  4. Server deduplicates by msg_id (Redis SETNX dedup:{msg_id} 86400)
  5. Server ACKs with sequence number → client stops retrying
  6. Client marks message as "sent" (single checkmark ✓)

RECIPIENT SIDE:
  1. Server pushes message to recipient's WebSocket
  2. Recipient client sends delivery receipt (ACK)
  3. If no ACK within 10s → server retries push (up to 3 times)
  4. If still no ACK → message stays in "pending delivery" state
  5. On next sync (reconnect), client pulls pending messages
  6. Client deduplicates by msg_id before rendering

DEDUPLICATION:
  Client maintains a local set of seen message IDs (last 10K messages).
  If a message arrives with an ID already in the set → silently drop.
  This handles: network retries, sync replays, server-side redelivery.

Client state per conversation:
  last_seen_sequence: 103

Incoming message arrives:
  sequence: 106  (gap! missing 104, 105)

Client action:
  1. Buffer message 106 (don't render yet — would break visual order)
  2. Request gap fill: GET /api/v1/conversations/{id}/messages?after_sequence=103&before_sequence=106
  3. Server returns messages 104, 105
  4. Client inserts 104, 105, 106 in order
  5. Update last_seen_sequence to 106

Why gaps happen:
  → Server delivered 104 and 105 but WebSocket dropped those frames
  → Network partition caused some messages to route via offline inbox
  → Multi-device: another device ACKed 104/105 but this device missed them

Optimization:
  If gap is small (1-2 messages), wait 500ms — they might arrive out of order.
  If gap persists after 500ms, trigger gap fill request.

States:
  PENDING    → Client created, not yet ACKed by server
  SENT       → Server ACKed (sequence assigned) — ✓
  DELIVERED  → Recipient device received — ✓✓
  READ       → Recipient opened conversation — ✓✓ (blue)
  FAILED     → Server rejected (content too long, user blocked, etc.)

Transitions:
  PENDING → SENT       : Server ACK received
  PENDING → FAILED     : Server NACK received (or timeout after 30s)
  SENT → DELIVERED     : Delivery receipt from ANY of recipient's devices
  DELIVERED → READ     : Read receipt from recipient

Rules:
  → States only move forward (DELIVERED never goes back to SENT)
  → DELIVERED means at least ONE device received it
  → READ is per-conversation, not per-message (read_up_to_sequence)
  → In groups: DELIVERED = all members received; READ = all members read
     (WhatsApp shows individual read status in group info)

For groups:
  → Track delivery/read per member (conversation_members.last_read_seq)
  → Show aggregate: "delivered to 45/50 members"
  → Individual status available on tap (like WhatsApp message info)

Client → Server (send message):
  Retry interval: 1s, 2s, 4s, 8s, 16s (exponential backoff)
  Max retries: 5
  After 5 failures: mark as FAILED, show error to user
  Idempotency: same msg_id on every retry → server deduplicates

Server → Recipient (deliver message):
  If recipient online: push via WebSocket, wait 10s for ACK
  If no ACK: retry push 3 times (1s, 2s, 4s)
  If still no ACK: assume connection is stale, mark offline
  Message goes to offline inbox → delivered on next sync

Server → Cassandra (persist):
  Retry with exponential backoff: 100ms, 200ms, 400ms
  Max retries: 10 (Cassandra is eventually available)
  If all retries fail: buffer in local WAL, retry on background thread
  Message is already delivered to recipient — persistence is best-effort
  (client devices are the ultimate source of truth for E2E encrypted messages)

Push notification (offline):
  Single attempt to APNs/FCM
  If rejected (invalid token): mark device token as stale
  If rate-limited: exponential backoff per device
  No retry for transient failures — next message will trigger a new push

Message arrives from Alice to group "Family" (8 members):

1. Chat Service assigns sequence: INCR seq:conv_family → 4821
2. Persist to Cassandra: INSERT INTO messages (conv_family, 4821, ...)
3. Fetch members: SELECT user_id FROM conversation_members WHERE conversation_id = conv_family
   → Returns: [alice, bob, carol, dave, eve, frank, grace, henry]
4. Remove sender: [bob, carol, dave, eve, frank, grace, henry] (7 recipients)
5. For each recipient, lookup connection:
   HGETALL user:bob:connections → {"iphone": "ws-042:conn_1a"}
   HGETALL user:carol:connections → {} (offline!)
   HGETALL user:dave:connections → {"android": "ws-117:conn_2b", "web": "ws-089:conn_3c"}
   ...

6. Online members: forward message to their WebSocket servers (parallel gRPC calls)
7. Offline members: enqueue push notification

Total time for 7 recipients: ~10ms (parallel lookups + parallel forwards)
All members receive within ~50ms of each other.

Message arrives from admin to "Tech News" channel (150K members):

1. Chat Service: assign sequence, persist to Cassandra, ACK sender → 30ms
2. Chat Service: produce to Kafka topic "group-fanout" → 5ms
   Payload: { conversation_id, sequence, message_id, sender_id }

3. Fan-out Service consumes from Kafka:
   a. Fetch member list in pages: 500 members per page
   b. For each page:
      - Batch-lookup connections: MGET user:{id}:connections for 500 users
      - Partition into online (has connection) and offline (no connection)
      - Online: batch-forward to WebSocket servers (grouped by server)
      - Offline: batch-enqueue push notifications
      - Apply backpressure: if downstream is slow, pause Kafka consumption
   c. Track progress: store last-processed page in Redis
      (enables resume on Fan-out Service crash)

4. Delivery timeline:
   - First 500 members: delivered within 100ms
   - All 150K members: delivered within 5-10 seconds
   - Push notifications: within 30 seconds (APNs/FCM batching)

5. Optimization: prioritize online members
   - Process online members first (they're waiting)
   - Offline members can wait — they'll sync on reconnect anyway

Problem: Celebrity posts to 1M-member channel
  → 1M connection lookups
  → 1M WebSocket pushes (for online members)
  → 500K push notifications (for offline members)
  → All within seconds

Mitigation layers:

Layer 1 — Fan-out on read for mega-channels
  Don't push to each member. Store message once.
  When member opens the channel → pull latest messages via sync.
  Push notification to a SAMPLE (10%) to drive engagement.
  → Reduces fan-out from 1M to ~100K (online members who have the channel open)

Layer 2 — Tiered delivery priority
  Members who have the channel open (foreground): immediate push
  Members who have the app open (background): push within 5s
  Members who are offline: push notification within 60s
  Members who muted the channel: no push, deliver on next sync only

Layer 3 — Batch and coalesce
  If celebrity posts 5 messages in 10 seconds:
  Don't send 5 separate pushes. Coalesce into one: "5 new messages in Tech News"
  Reduces push notification volume by 5×

Layer 4 — Dedicated fan-out workers per hot channel
  Detect hot channels (>10K members + >1 msg/min)
  Assign dedicated Kafka partition + dedicated consumer group
  Prevents hot channel from starving delivery to small groups

Rule 1: Snapshot membership at send time
  When a message is accepted, the member list at that moment is the
  delivery target. If Bob leaves 1 second later, he still gets the message
  (it was sent while he was a member).

Rule 2: New members don't see pre-join messages (by default)
  When Carol joins, her "visible_from_sequence" is set to the current
  sequence. She only sees messages with sequence >= visible_from_sequence.
  (Telegram shows history; WhatsApp doesn't. This is a product decision.)

Rule 3: Removed members stop receiving immediately
  On removal: delete from conversation_members + revoke E2E keys.
  Any in-flight messages already delivered are fine (they were a member).
  No new messages will be delivered.

Implementation:
  Fan-out Service checks membership at delivery time:
  IF user is no longer a member AND message.sequence > user.left_at_sequence
  THEN skip delivery (they left after this message was sent — don't deliver)

Edge case: user leaves and rejoins within the fan-out window
  → Use left_at_sequence and rejoined_at_sequence to determine visibility

Stored state:
  conversation_members.last_read_seq = 4815 (Bob's last read in "Family")
  conversations.last_message_seq = 4821 (latest message in "Family")

Unread count = 4821 - 4815 = 6 messages

When Bob opens the conversation:
  UPDATE conversation_members SET last_read_seq = 4821
    WHERE conversation_id = conv_family AND user_id = bob
  → Unread count becomes 0

When Bob sends a read receipt:
  Server updates last_read_seq
  Server forwards read receipt to sender(s) of unread messages
  → Sender sees blue checkmarks appear

Why this works at scale:
  → No COUNT(*) query needed — just arithmetic
  → Single row update on read (not N updates for N unread messages)
  → Works for groups of any size
  → Muted conversations: still track last_read_seq but don't show badge

States:
  ONLINE       → User has an active WebSocket connection
  OFFLINE      → No active connection (show "last seen" timestamp)
  RECENTLY     → Connection dropped <5 min ago (grace period before showing offline)

Transitions:
  OFFLINE → ONLINE    : WebSocket connection established + authenticated
  ONLINE → RECENTLY   : WebSocket disconnects (network switch, app backgrounded)
  RECENTLY → ONLINE   : Reconnects within 5 minutes (no visible change to contacts)
  RECENTLY → OFFLINE  : 5-minute grace period expires without reconnection

Why the RECENTLY state?
  → Users switch from WiFi to cellular constantly (brief disconnect)
  → App goes to background on iOS (WebSocket may close)
  → Without grace period, users would flicker online/offline every few minutes
  → WhatsApp uses ~5 min grace; Telegram uses ~30 seconds

Key: presence:{user_id}
Value: { "status": "online", "last_seen": 1716300000, "device": "iphone" }
TTL: 60 seconds (refreshed by heartbeat)

On connect:
  SET presence:{user_id} '{"status":"online","last_seen":null}' EX 60
  PUBLISH presence_updates:{user_id} "online"

On heartbeat (every 30s):
  EXPIRE presence:{user_id} 60  // refresh TTL

On disconnect:
  SET presence:{user_id} '{"status":"recently","last_seen":1716300000}' EX 300
  // After 5 min TTL expires → key deleted → user is fully offline

On TTL expiry (no reconnect within grace period):
  Key auto-deletes → presence lookup returns null → treat as offline
  No explicit "offline" event needed — absence of key = offline

Design principles:
  1. NEVER persist to database or queue
  2. NEVER retry on failure (if it doesn't arrive, that's fine)
  3. Auto-expire after 5 seconds (no explicit "stopped typing" needed)
  4. Throttle: max 1 typing event per 3 seconds per user per conversation
  5. Only send to users who have the conversation OPEN (foreground)

Flow:
  Alice starts typing in conv_abc:
  1. Client sends typing frame: { type: "typing", conv: "conv_abc", action: "started" }
  2. WS Server receives → forwards to Presence Service (in-memory)
  3. Presence Service checks: who has conv_abc open right now?
     → Subscribers: [bob (ws-042), carol (ws-117)]
  4. Forward typing indicator to Bob and Carol's WS servers
  5. Bob and Carol's clients show "Alice is typing..."
  6. After 5 seconds with no renewal → clients auto-hide indicator

Throttling:
  Client-side: don't send more than 1 event per 3 seconds
  Server-side: rate limit per (user_id, conversation_id) → drop excess
  → At 500M DAU with 10% actively typing: 50M × 1/3s = 16.7M events/sec
  → With subscriber filtering: most events go to 1-5 recipients
  → Actual delivery load: ~50M deliveries/sec (manageable across 200 WS servers)

Group typing:
  Show up to 3 names: "Alice, Bob, and 2 others are typing..."
  Don't fan out typing to ALL group members — only those with chat open
  For 200K-member channel: maybe 500 have it open → 500 deliveries per typing event

How it works:

1. ONLINE STATUS (real-time, no DB writes):
   User connects → Redis key exists → they're online
   Determined by: presence:{user_id} key with TTL in Redis
   Cost: zero additional writes (already maintained for routing)

2. LAST SEEN (persistent, written only on disconnect):
   User disconnects → write to LastSeen table:
   UPDATE last_seen SET timestamp = NOW() WHERE user_id = ?
   Use conditional write: only update if new timestamp > existing
   (prevents race conditions between multiple servers)

3. QUERYING (when Alice opens chat with Bob):
   Step 1: Check Redis presence:{bob} → if exists → "Online" (done!)
   Step 2: If not in Redis → query LastSeen table → "Last seen 5 min ago"
   
   This means:
   → Online users: answered from Redis (sub-ms, no DB hit)
   → Offline users: answered from LastSeen table (single read)
   → No polling, no periodic writes, no write amplification

4. EDGE CASE: Server crashes without writing disconnect
   → Redis TTL expires (60s) → user appears offline
   → LastSeen table shows the PREVIOUS disconnect time (slightly stale)
   → When user reconnects to a new server, the old server's crash
     is detected and LastSeen is backfilled from the TTL expiry time
   → Acceptable staleness: at most 60 seconds off

Privacy settings (stored in user profile, cached in Redis):
  user:{user_id}:privacy:last_seen → "everyone" | "contacts" | "nobody"

On presence query:
  1. Fetch target user's privacy setting
  2. If "everyone" → return last_seen timestamp
  3. If "contacts" → check if requester is in target's contact list
  4. If "nobody" → return null (show nothing)

Reciprocity rule (WhatsApp behavior):
  If Alice hides her last seen from Bob, Alice also can't see Bob's last seen.
  This prevents asymmetric information and reduces privacy complaints.

Implementation:
  On presence subscription: check privacy ONCE (on subscribe)
  Cache the result for the subscription lifetime
  If privacy setting changes: invalidate all active subscriptions for that user
  → Subscribers re-check on next presence event

Redis memory for presence:
  500M users × 100 bytes per presence entry = 50 GB
  But only online users need entries: 50M × 100 bytes = 5 GB
  → Fits in a single Redis Cluster (6 nodes × 8GB each)

Presence event throughput:
  Status changes: ~30K/sec (online/offline transitions)
  Typing indicators: ~16.7M/sec (generated) → ~50M/sec (delivered)
  Heartbeats: 50M connections / 30s = 1.67M/sec

  → Typing dominates. But typing is fire-and-forget with no persistence.
  → The Presence Service is essentially a real-time event router, not a database.

Subscription tracking:
  "Who is subscribed to Bob's presence?"
  Redis SET: presence_subscribers:{user_id} → set of subscriber user_ids
  Average: 2-3 subscribers per online user (people who have chat open with them)
  50M online × 3 subscribers = 150M subscription entries
  At 32 bytes per entry: ~5 GB additional Redis memory