Core Architecture

# Every document has metadata fields:

{
  "_index": "products",          ← which index this doc belongs to
  "_id": "abc123",               ← unique identifier (auto-generated or explicit)
  "_version": 3,                 ← incremented on every update
  "_source": {                   ← the actual JSON you indexed
    "name": "Running Shoes",
    "price": 129.99,
    "category": "footwear",
    "description": "Lightweight trail running shoes"
  }
}

Key properties:
  • Documents are JSON — nested objects, arrays, all valid
  • Documents are immutable — you cannot modify a document in place
  • Updates = delete old version + reindex new version (internally)
  • _id is unique within an index (not globally)
  • _source stores the original JSON (retrievable, but not searched directly)

Index: "products"
  ├── Mapping (schema): defines field types and analyzers
  ├── Settings: number_of_shards=5, number_of_replicas=1
  ├── Aliases: "products-live" → "products-v2"
  └── Documents: millions of product JSON objects

Common patterns:
  • One index per entity type: products, users, orders
  • Time-based indices: logs-2024-01-15, logs-2024-01-16
  • Versioned indices: products-v1, products-v2 (swap via alias)

Index naming rules:
  • Lowercase only
  • No special characters (except hyphens and underscores)
  • Cannot start with - or _
  • Max 255 characters

# Step 1: Create new index with updated mapping
PUT /products-v2
{ "mappings": { ... }, "settings": { ... } }

# Step 2: Reindex all documents from old to new
POST /_reindex
{ "source": { "index": "products-v1" }, "dest": { "index": "products-v2" } }

# Step 3: Atomic alias swap (clients never notice)
POST /_aliases
{
  "actions": [
    { "remove": { "index": "products-v1", "alias": "products" } },
    { "add":    { "index": "products-v2", "alias": "products" } }
  ]
}

# Clients always query "products" alias — zero downtime
# Old index can be deleted after verification

Index: "products" — 5 primary shards, 1 replica

Node 1 (data)          Node 2 (data)          Node 3 (data)
┌──────────────┐       ┌──────────────┐       ┌──────────────┐
│  P0  │  P3   │       │  P1  │  P4   │       │  P2          │
│      │       │       │      │       │       │              │
│  R1  │  R4   │       │  R2  │  R0   │       │  R3          │
└──────────────┘       └──────────────┘       └──────────────┘

P = Primary shard    R = Replica shard
• Each primary lives on exactly one node
• Replicas are placed on DIFFERENT nodes than their primary
• Queries can be served by either primary or replica
• Writes always go to the primary first, then replicate

Document routing:
  shard = hash(_routing) % number_of_primary_shards
  default _routing = document _id

  This is WHY you can't change shard count after creation —
  the routing formula would assign documents to wrong shards.

Index: "orders" — 3 primaries, 2 replicas (number_of_replicas=2)

Each primary shard has 2 copies → 3 primaries + 6 replicas = 9 total shards

Node 1            Node 2            Node 3
┌────────────┐    ┌────────────┐    ┌────────────┐
│ P0         │    │ P1         │    │ P2         │
│ R1-copy1   │    │ R0-copy1   │    │ R0-copy2   │
│ R2-copy1   │    │ R2-copy2   │    │ R1-copy2   │
└────────────┘    └────────────┘    └────────────┘

Write path:
  1. Client sends index request → coordinating node
  2. Routed to primary shard (P0 on Node 1)
  3. Primary indexes document, writes to translog
  4. Primary forwards to ALL replicas (R0-copy1, R0-copy2)
  5. Replicas acknowledge → primary acknowledges client

Read path:
  • Any copy (primary OR replica) can serve a read
  • Coordinating node round-robins across available copies
  • More replicas = more read throughput (linear scaling)

# Dedicated master node (no data, no ingest)
node.roles: [master]
# Lightweight — needs minimal CPU/disk, moderate RAM for cluster state

# Dedicated data node (the workhorse)
node.roles: [data]
# Heavy — needs lots of disk, RAM, and CPU for indexing/search

# Dedicated ingest node
node.roles: [ingest]
# Moderate — CPU for pipeline processing

# Coordinating-only node (no roles = coordinating only)
node.roles: []
# Acts as a smart load balancer — scatters/gathers queries

# Small cluster (mixed roles — all nodes do everything)
node.roles: [master, data, ingest]
# Fine for < 5 nodes, but doesn't scale well

Production topology (typical):
  3 dedicated master nodes    (m5.large — small, stable)
  N dedicated data nodes      (i3.2xlarge — big disk, lots of RAM)
  2 coordinating-only nodes   (c5.xlarge — CPU for merging results)
  Optional: 1-2 ingest nodes  (c5.xlarge — CPU for pipelines)

GET /_cluster/health

{
  "cluster_name": "production",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 5,
  "number_of_data_nodes": 3,
  "active_primary_shards": 42,
  "active_shards": 78,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 6,          ← these are the problem
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 92.86
}

# Diagnosis: 6 unassigned shards (replicas, since status is yellow not red)
# Likely cause: only 3 data nodes but some indices need replicas on 4+ nodes

# To find which shards are unassigned:
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state

# To see why a shard can't be allocated:
GET /_cluster/allocation/explain

Document indexed → In-memory buffer → [refresh] → New segment (searchable)
                                    ↓
                              Translog (durability)
                                    ↓
                              [flush] → Segment on disk, translog cleared

Timeline of a write:
  t=0ms    Document arrives, written to in-memory buffer + translog
  t=0ms    Document is NOT yet searchable
  t=1000ms Refresh happens (every 1 second by default)
           → In-memory buffer → new immutable segment
           → Document is NOW searchable
  t=30min  Flush happens (or translog size threshold)
           → Segment fsync'd to disk
           → Translog cleared

This 1-second gap is why ES is "near-real-time":
  • You index a document at t=0
  • It becomes searchable at t=1s (after next refresh)
  • NOT instantly searchable like a database INSERT

Before merge:
  Shard P0: [seg-1: 50MB] [seg-2: 12MB] [seg-3: 8MB] [seg-4: 3MB] [seg-5: 1MB]
            5 segments, some with deleted docs marked

After merge (background process):
  Shard P0: [seg-1: 50MB] [seg-6: 22MB]
            2 segments, deleted docs physically removed

Merge behavior:
  • Automatic — Lucene's TieredMergePolicy handles it
  • Merges small segments into larger ones
  • Physically removes deleted documents (reclaims disk)
  • CPU and I/O intensive — runs in background threads
  • max_num_segments: 1 via _forcemerge (only for read-only indices!)

Key insight: "deleting" a document just marks it in a bitset.
The bytes aren't reclaimed until a merge happens.

Client → Coordinating Node → Primary Shard → Replica Shards

Detailed flow on the primary shard:
  1. Document arrives via Bulk API or single index request
  2. Ingest pipeline runs (if configured): enrichment, parsing
  3. Document is analyzed: text fields → tokens via analyzer
  4. Written to in-memory buffer (Lucene IndexWriter)
  5. Written to translog (append-only, sequential write — fast)
  6. Acknowledgment sent to client (document is durable but NOT searchable)
  7. [After refresh_interval] Buffer → new Lucene segment (now searchable)
  8. [After flush] Segment fsync'd, translog cleared

Performance levers:
  • refresh_interval: "30s"     — less frequent refresh = fewer segments = faster indexing
  • refresh_interval: "-1"      — disable refresh entirely during bulk load
  • translog.durability: "async" — don't fsync translog on every request (faster, less safe)
  • number_of_replicas: 0       — no replica writes during bulk load

POST /_bulk
{"index": {"_index": "products", "_id": "1"}}
{"name": "Running Shoes", "price": 129.99}
{"index": {"_index": "products", "_id": "2"}}
{"name": "Trail Boots", "price": 189.99}
{"index": {"_index": "products", "_id": "3"}}
{"name": "Sandals", "price": 49.99}

Why bulk over single requests:
  • Single: 1000 documents = 1000 HTTP round trips
  • Bulk: 1000 documents = 1 HTTP request
  • 10-100x faster for batch operations

Bulk sizing guidelines:
  • Start with 5-15 MB per bulk request
  • Too small: overhead per request dominates
  • Too large: memory pressure, long GC pauses
  • Measure and adjust: watch for rejected requests (thread pool full)
  • Optimal batch size varies by document size and cluster capacity

# Merge all segments in an index down to 1 segment
POST /logs-2024-01-15/_forcemerge?max_num_segments=1

# When to use:
#   ✅ Time-based indices that are no longer receiving writes (yesterday's logs)
#   ✅ After bulk load + setting index to read-only
#   ❌ NEVER on actively-written indices (blocks new segments, wastes I/O)

# Benefits of force merge on read-only indices:
#   • Fewer segments = faster searches (less merging at query time)
#   • Deleted documents physically removed (reclaims disk)
#   • Reduced file descriptor usage
#   • Better OS page cache utilization