Partitioning

Single node: 10,000 writes/sec (max)
Requirement: 100,000 writes/sec

Option 1: Bigger machine (vertical scaling)
  → Maybe 20,000 writes/sec with 2x hardware
  → Still not enough. And 10x hardware doesn't exist.

Option 2: 10 partitions (horizontal scaling)
  → Each partition handles 10,000 writes/sec
  → Total: 10 × 10,000 = 100,000 writes/sec ✅
  → Need more? Add more partitions.

The key: a good partition key distributes writes EVENLY.
If 90% of writes go to 1 partition → you're back to square one.

Write: INSERT user { id: "user_42", name: "Alice", ... }

Step 1: Compute shard
  shard = hash("user_42") % 4
  hash("user_42") = 2847291
  2847291 % 4 = 3
  → Route to Shard 3

Step 2: Shard 3 handles the write
  → Updates its local index
  → Writes to its local WAL
  → Replicates to its followers

Step 3: Read for user_42
  → Same computation: hash("user_42") % 4 = 3
  → Route to Shard 3
  → Single-shard read (fast)

Cross-shard query: "Find all users named Alice"
  → Must query ALL 4 shards (scatter)
  → Merge results (gather)
  → Slow — this is the cost of sharding

Problem: Celebrity post gets 1M likes/minute
  All writes: INSERT INTO likes (post_id, user_id, ...)
  post_id = "post_viral_123"
  hash("post_viral_123") % 10 = shard 7
  → Shard 7 gets 1M writes/min, others get ~0

Solution: Salt the key
  salted_key = "post_viral_123_" + random(0, 9)
  → "post_viral_123_0" → shard 2
  → "post_viral_123_3" → shard 5
  → "post_viral_123_7" → shard 1
  → Writes spread across ~10 shards

Reading like count:
  → Query all 10 salted keys: post_viral_123_0 through _9
  → SUM the counts
  → 10 parallel reads instead of 1, but no hot partition

Alice posts a photo. Alice has 500 followers.

At WRITE time:
  1. Store the post in posts table
  2. Look up Alice's 500 followers
  3. Write the post_id into each follower's feed:
     INSERT INTO feed (user_id, post_id) VALUES
       (follower_1, post_123),
       (follower_2, post_123),
       ...
       (follower_500, post_123)

  Write cost: 1 post + 500 feed entries = 501 writes
  Read cost: SELECT * FROM feed WHERE user_id = ? ORDER BY time
             → Single partition read, pre-sorted. FAST.

When follower opens their feed:
  → Data is already there, pre-computed
  → One read, one partition, instant

Alice posts a photo. Alice has 500 followers.

At WRITE time:
  1. Store the post in posts table
  Write cost: 1 write. Done.

When follower opens their feed:
  1. Look up who they follow: [Alice, Bob, Charlie, ...]
  2. Fetch recent posts from EACH followed user:
     SELECT * FROM posts WHERE author_id = 'alice' ORDER BY time LIMIT 20
     SELECT * FROM posts WHERE author_id = 'bob' ORDER BY time LIMIT 20
     ...
  3. Merge and sort all results by time
  4. Return top 20

  Read cost: N queries (one per followed user) + merge + sort
  If user follows 500 people → 500 queries per feed load. SLOW.

Rule: Use fan-out on write for normal users, fan-out on read for celebrities.

Alice posts (500 followers → fan-out on write):
  → Write post_id into 500 follower feeds
  → Followers see it instantly in their pre-built feed

Taylor Swift posts (100M followers → fan-out on read):
  → Store post once
  → Do NOT write to 100M feeds (would take minutes, waste storage)
  → When a follower loads their feed:
    1. Read pre-built feed (fan-out on write posts)
    2. Fetch latest posts from followed celebrities (fan-out on read)
    3. Merge the two lists
    → Slightly slower read, but avoids 100M writes

Threshold: typically ~5,000-10,000 followers
  Below → fan-out on write
  Above → fan-out on read

Update-in-place (traditional):
  UPDATE accounts SET balance = 300 WHERE id = 42
  → Random disk seek to find row
  → Lock the row
  → Update in place
  → Update index
  → Write WAL entry
  → Slow: random I/O + locking

Append-only:
  INSERT INTO account_events (account_id, type, amount, balance, timestamp)
  VALUES (42, 'withdrawal', 200, 300, NOW())
  → Sequential disk append (fastest I/O pattern)
  → No locking (no row to lock)
  → No index update on existing data
  → Fast: sequential I/O, no contention

Current balance: SELECT balance FROM account_events
                 WHERE account_id = 42
                 ORDER BY timestamp DESC LIMIT 1

Full history: SELECT * FROM account_events
              WHERE account_id = 42
              ORDER BY timestamp

Traffic: 500K writes/sec (posts, likes, comments, follows)
Users: 500M total, 50M DAU
Celebrities: 10K users with > 100K followers

1. WRITE SHARDING
   Posts table: sharded by hash(userId) across 50 partitions
   Each partition handles ~10K writes/sec (within capacity)
   All of a user's posts on one shard → fast user profile reads

2. HOT PARTITION AVOIDANCE
   Likes on viral posts: salted key
   "like:post_viral_123" → "like:post_viral_123_{0-9}"
   Writes spread across 10 shards instead of 1
   Like count: SUM across all 10 salted keys (async counter)

3. FAN-OUT STRATEGY (Hybrid)
   Normal user posts (< 5K followers): fan-out on write
   → Post written + 5K feed entries = 5,001 writes
   → Followers see it instantly in pre-built feed

   Celebrity posts (> 100K followers): fan-out on read
   → Post written once
   → Feed service merges pre-built feed + celebrity posts at read time

4. APPEND-ONLY DESIGN
   All writes go to Kafka first (append-only log)
   Kafka partitioned by userId (same user → same partition → ordering)
   Consumers read from Kafka and write to:
   → Posts DB (sharded PostgreSQL)
   → Feed cache (Redis, pre-built feeds)
   → Search index (Elasticsearch)
   → Analytics (ClickHouse)

Flow:
  Client → API → Kafka (append) → Consumers → [DB, Cache, Search, Analytics]
  ↑ Fast (Kafka append)              ↑ Async (decoupled from user request)