LinkedIn Is Moving Beyond Kafka — A Deep Dive into Northguard

Before we talk about replacing Kafka, we need to understand why it became irreplaceable in the first place. Kafka solved a problem that every growing company eventually hits: how do you move data between systems in real time without losing anything?

Before Kafka, companies stitched together point-to-point integrations. Service A writes to a database, Service B polls that database, Service C gets a webhook. It works until it doesn't — and it stops working the moment you need ordering guarantees, replay capability, or more than a handful of consumers.

Service A ──→ Database ←── Service B (polling)
Service A ──→ Webhook  ──→ Service C
Service A ──→ Queue    ──→ Service D
Service B ──→ REST API ──→ Service E

Problems:
  • No single source of truth for event ordering
  • Adding a new consumer means modifying the producer
  • No replay — if Service C was down, those events are gone
  • Scaling consumers independently is nearly impossible

Kafka introduced a deceptively simple abstraction: an append-only, partitioned, replicated log. Producers write events to the end of the log. Consumers read from wherever they left off. The log retains data for a configurable period (or forever). Multiple consumers can read the same data independently, at their own pace.

This model was so effective that Kafka became the backbone of event-driven architecture at thousands of companies. Netflix uses it for real-time recommendations. Uber uses it for trip event processing. Banks use it for transaction streams. It's everywhere.

When LinkedIn created Kafka in 2010, they had around 90 million members. The platform handled activity feeds, metrics pipelines, and log aggregation. Kafka was designed for this workload and it handled it well.

Fast forward to 2026. LinkedIn has over 1.2 billion members. The platform processes more than 32 trillion messages per day across 150+ Kafka clusters containing over 400,000 topics. To put that in perspective: that's roughly 370 million messages per second, sustained, 24/7.

                    2010                    2026
Members:            90 million              1.2 billion    (13x)
Daily messages:     millions                32 trillion    (millions-x)
Clusters:           single digits           150+
Topics:             hundreds                400,000+
Partitions:         thousands               millions

The architecture that worked at 90M members
doesn't necessarily work at 1.2B members.
That's not a flaw — that's physics.

The important thing to understand is that LinkedIn didn't wake up one morning and decide Kafka was bad. They spent over a decade optimizing it. They contributed heavily to the open-source project. They built custom tooling around it. They squeezed every last drop of performance out of the architecture. And then they hit walls that no amount of tuning could fix.

LinkedIn's engineers identified three fundamental design constraints in Kafka that couldn't be solved with configuration changes or incremental improvements. These aren't bugs — they're consequences of architectural decisions that were perfectly reasonable at Kafka's original scale.

Wall 1: The Partition Is the Unit of Everything

In Kafka, a partition is simultaneously the unit of storage, replication, consumption, and rebalancing. This tight coupling is elegant at moderate scale but becomes a liability at extreme scale.

When a partition grows to hundreds of gigabytes (common for high-throughput topics), every operation that involves moving that partition becomes expensive. Adding a broker? You need to copy entire partitions — potentially terabytes of data — across the network. A broker fails? Its partition replicas need to be fully reconstructed elsewhere.

Topic: user-activity (high throughput)
  Partition 0: 340 GB on Broker A
  Partition 1: 280 GB on Broker B
  Partition 2: 510 GB on Broker C  ← hot partition
  Partition 3: 190 GB on Broker D

Scenario: Add Broker E to the cluster

  Step 1: Decide which partitions to move (manual or semi-auto)
  Step 2: Copy selected partitions to Broker E
          → Partition 2 alone is 510 GB
          → At 100 MB/s network throughput: ~85 minutes just for one partition
  Step 3: During copy: Broker C has elevated disk I/O + network usage
  Step 4: Consumers on Partition 2 may see latency spikes
  Step 5: If the copy fails midway, start over

Total time for a full rebalance across the cluster: hours to days

Wall 2: Centralized Metadata Coordination

Kafka originally used ZooKeeper for metadata management, and more recently moved to KRaft (Kafka Raft) to eliminate the ZooKeeper dependency. In both models, a single controller node is responsible for all metadata decisions: which broker leads which partition, where replicas live, handling broker failures.

At LinkedIn's scale — millions of partitions across 150 clusters — this single controller becomes a coordination bottleneck. Leader elections pile up. Topic creation slows down. Broker failure recovery takes longer because the controller is processing a queue of metadata changes.

Single Kafka Controller responsibilities:
  ├── Track state of every partition (millions)
  ├── Elect new leaders when brokers fail
  ├── Process topic creation/deletion requests
  ├── Handle consumer group rebalances
  └── Propagate metadata updates to all brokers

At 400,000 topics × avg 10 partitions = 4,000,000 partition states

When Broker X fails:
  → Controller must reassign all partitions led by Broker X
  → While processing reassignments, other operations queue up
  → New topic creation? Wait.
  → Another broker failure? Queue behind the first one.
  → Metadata propagation to 100+ brokers? Delayed.

The controller isn't slow — it's doing too many things sequentially.

Wall 3: Static Partition Assignment Creates Skew

Kafka assigns partitions to brokers at creation time. Once assigned, a partition stays on that broker unless explicitly moved. Over time, this creates imbalances: some brokers end up with more data, more traffic, or both. The industry calls this "hot spots."

At small scale, you fix this with manual rebalancing. At LinkedIn's scale, with millions of partitions and constantly shifting traffic patterns, manual rebalancing is like trying to level a swimming pool with a teaspoon.

If the root problem is that partitions are too large and indivisible, the solution is to make them smaller and divisible. That's exactly what log striping does.

In Northguard, a logical partition still exists as a concept — producers write to it, consumers read from it, ordering is preserved. But under the hood, the data isn't stored as one contiguous blob on a single broker. Instead, it's broken into fixed-size segments (roughly 1 GB each) called "stripes," and those stripes are distributed across multiple brokers.

KAFKA — One partition, one broker:

  Broker A
  ┌─────────────────────────────────────────┐
  │  Partition 0                            │
  │  [offset 0 ─────────────── offset 50M]  │
  │  Total: 340 GB, all on Broker A         │
  └─────────────────────────────────────────┘

NORTHGUARD — One logical partition, many stripes:

  Logical Partition 0 (same offsets, same ordering)
  ┌────────┐ ┌────────┐ ┌────────┐     ┌────────┐
  │Stripe 1│ │Stripe 2│ │Stripe 3│ ... │Stripe N│
  │  1 GB  │ │  1 GB  │ │  1 GB  │     │  1 GB  │
  │Broker A│ │Broker C│ │Broker B│     │Broker D│
  └────────┘ └────────┘ └────────┘     └────────┘

  Consumer sees: one continuous ordered log (unchanged API)
  System sees: 340 independent 1 GB chunks across the cluster

Why 1 GB Changes Everything

The choice of a small, fixed stripe size has cascading benefits across every operational dimension:

The second major architectural change in Northguard addresses the centralized controller problem. Instead of one node managing all metadata, Northguard distributes metadata across multiple independent groups, each running the Raft consensus protocol.

Quick Primer: What Is Raft?

Raft is a consensus algorithm that allows a group of nodes to agree on a shared state, even if some nodes fail. It works by electing a leader within the group. The leader accepts writes, replicates them to followers, and commits once a majority acknowledges. If the leader dies, the remaining nodes elect a new one — typically within milliseconds.

The key property: as long as a majority of nodes in the group are alive, the group continues operating correctly. This is much more resilient than a single-node controller.

KAFKA (KRaft mode):

  ┌──────────────────────────────────────────┐
  │         Single Controller Quorum         │
  │                                          │
  │  Manages ALL metadata for ALL topics:    │
  │  • 400,000 topics                        │
  │  • 4,000,000 partitions                  │
  │  • All leader elections                  │
  │  • All broker state                      │
  │                                          │
  │  Throughput ceiling: one quorum's worth  │
  └──────────────────────────────────────────┘

NORTHGUARD:

  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
  │ Raft Group 1 │  │ Raft Group 2 │  │ Raft Group 3 │  ...
  │              │  │              │  │              │
  │ Topics A–D   │  │ Topics E–H   │  │ Topics I–L   │
  │ 3 replicas   │  │ 3 replicas   │  │ 3 replicas   │
  │              │  │              │  │              │
  │ Independent  │  │ Independent  │  │ Independent  │
  │ leader       │  │ leader       │  │ leader       │
  └──────────────┘  └──────────────┘  └──────────────┘

  Each group handles a subset of topics independently.
  Total metadata throughput = sum of all groups.
  Failure in Group 2 doesn't affect Groups 1 or 3.

Why Sharding Metadata Matters

The benefit isn't just fault isolation — it's horizontal scalability of the control plane itself. Need to handle more topics? Add more Raft groups. Each group operates independently, so the total metadata throughput scales linearly with the number of groups.

Compare this to Kafka's approach: even with KRaft replacing ZooKeeper, you still have a single quorum handling all metadata. You can make that quorum faster (better hardware, optimized code), but you can't shard it. It's a vertical scaling ceiling on the control plane.

Building a better streaming platform is one thing. Migrating 32 trillion daily messages to it without downtime is another thing entirely. This is arguably the most impressive part of the whole effort — and the part most engineers underestimate.

LinkedIn's solution is Xinfra: a virtualized Pub/Sub abstraction layer that sits between applications and the underlying streaming infrastructure. Applications talk to Xinfra through a unified API. Xinfra decides whether to route messages to Kafka, Northguard, or both.

The Four-Phase Migration

LinkedIn follows a careful, reversible migration strategy. Each phase builds confidence before moving to the next:

Application code (unchanged across all phases):

  xinfra.produce("user-events", event);
  xinfra.consume("user-events", handler);

Phase 1 — Xinfra config:
  user-events → backend: kafka

Phase 2 — Xinfra config:
  user-events → backend: kafka + northguard (dual-write)

Phase 3 — Xinfra config:
  user-events → write: kafka + northguard
                read:  kafka (primary) + northguard (shadow)

Phase 4 — Xinfra config:
  user-events → backend: northguard

Rollback at any phase:
  user-events → backend: kafka

As of early 2026, roughly 90% of LinkedIn's applications have been migrated to the Xinfra API. The hardest part — decoupling applications from the Kafka client — is largely complete. The remaining work is routing individual topics from Kafka to Northguard, which is a configuration change per topic.

Let's put the two systems side by side across the dimensions that matter most for operating a streaming platform at scale.

Where Each System Excels

Let's be direct: unless you work at one of maybe 20 companies on Earth, this changes nothing about your technology choices today. Northguard is not available outside LinkedIn. You cannot download it, deploy it, or buy it as a managed service.

But the ideas behind it are worth understanding, because they reveal where the streaming ecosystem is heading — and because the migration strategy (Xinfra) is a pattern you can apply regardless of scale.

Alternatives You Can Actually Use Today

If you're hitting Kafka's limits but can't build your own Northguard, there are production-ready alternatives that address some of the same pain points:

The Transferable Lesson: Build Abstraction Layers Early

The most practical takeaway from LinkedIn's story isn't about Northguard — it's about Xinfra. By wrapping their Kafka dependency behind an abstraction layer, they made the underlying infrastructure swappable. This is a pattern you can apply at any scale:

// ❌ Tight coupling — every service imports the Kafka client directly
import { Kafka } from 'kafkajs';
const kafka = new Kafka({ brokers: ['broker1:9092'] });
const producer = kafka.producer();
await producer.send({ topic: 'user-events', messages: [{ value: payload }] });

// ✅ Abstraction layer — services import your messaging interface
import { messageBus } from '@/lib/messaging';
await messageBus.publish('user-events', payload);

// The implementation behind messageBus can be:
//   - KafkaJS today
//   - Redpanda tomorrow
//   - An in-memory bus for testing
//   - A dual-write setup during migration
// No application code changes required.

LinkedIn's move beyond Kafka is a landmark moment in data infrastructure. Not because Kafka is dying — it isn't — but because it demonstrates what happens when a system designed for one era of scale meets the demands of the next.