The million dollar question when setting up your first Elasticsearch cluster: sizing. It feels like it should have an obvious answer. How many nodes? How much storage? What’s the limit? But the reality is more complex. There isn’t one hard ceiling. Elastic has demonstrated a single node querying 1 PiB of data. So we know the cluster scales. But what doesn’t scale automatically is the configuration around it.
There are three specific constraints that quietly degrade cluster health if you leave them unmanaged: shard size, shard count per node, and storage tier. Get those right and your cluster grows cleanly. Get them wrong (or miss them entirely) and the symptoms, like slow queries, failed allocations, and bills that grow faster than your data, will show up weeks or months after the decisions that caused them.
What is a shard, and why does its size matter?
Before our sizing rules make sense, you need to understand the underlying model.
Elasticsearch distributes data across shards.
Think of a shard as a filing drawer in a large archive. Each drawer holds a self-contained set of documents. When you run a query, Elasticsearch opens every relevant drawer simultaneously and merges the results. More drawers means more parallel work — which sounds good, until you realize that each drawer has overhead. Your master node has to track where every drawer is, what it contains, and whether it’s healthy or not. A thousand tiny drawers with a handful of files each is much harder to manage than fifty well-organized ones, even when the total document count is the same.
So here’s the rule: Aim to keep each shard between 10 GB and 50 GB, and under 200 million documents.
What happens if you don’t keep shards in that range?
Shards that are too small multiply overhead. Each shard consumes heap on the master node, and each query fans out to every shard in the index. A cluster with a thousand 100 MB shards carrying the same data as twenty 5 GB shards will be slower and harder to manage.
Shards that are too large slow query execution because more work happens inside a single Lucene segment. Failure recovery gets painful too. Elasticsearch recovers shards one at a time. A 200 GB shard on a failed node is a long, unscheduled wait.
What about document count?
Document count is a separate constraint. A shard full of short access log lines can stay under 50 GB and still carry hundreds of millions of documents. A shard full of enriched records might hit 50 GB at 30 million records.
To manage this, in your index lifecycle management (ILM) rollover policy, set both limits and let whichever fires first take control:
"rollover": {
"max_primary_shard_size": "50gb",
"max_primary_shard_docs": 200000000,
"max_age": "30d"
}Run the following command to see what your shards look like right now:
GET /_cat/shards?v=true&s=store:desc
This will sort your shards by store in descending order. Anything regularly over 50 GB needs either a tighter rollover size threshold or more primary shards defined at index creation time.
An aside: There’s one rule you may have seen that you can stop applying: no more than 20 shards per GB of heap. This guideline was deprecated in Elasticsearch v8.3 after internal optimizations (off-heap data structures, compressed cluster state, more compact metadata serialization) reduced per-shard overhead significantly. The current guidance is simpler: watch shard size, watch shard count, and set both rollover limits.
The shard budget you don’t know you’re spending
Most newcomers don’t realize they’re accumulating shards at all. Index lifecycle management (ILM) creates them on a schedule whether you’re tracking them or not.
Every non-frozen data node has a soft ceiling of 1,000 shards. With a daily rollover policy, 5 primary shards, and 1 replica, that’s 10 new shards per day. That means one node will hit the ceiling in 100 days without a single configuration change. Elasticsearch then stops allocating new shards to that node, and you start seeing allocation failures.
The fix for this depends on what’s driving the accumulation:
Widen the rollover window. If your shards rarely reach 50 GB before the time trigger fires, you’re rolling over more often than your data warrants. Switching from daily to weekly or monthly rollovers produces fewer, larger shards and extends your runway considerably.
Reduce primary shards per index. Small daily ingest volumes rarely need 5 primary shards. One or two is usually enough. For indices that are already over-sharded, shrinking can rebalance them.
Add nodes. If ingest volume genuinely requires frequent rollover at full shard count, distributing across more nodes is the answer.
Run the following command to see where you stand:
GET _cat/allocation?h=node,shards&v
One additional number worth knowing for master node planning: budget roughly 1 GB of heap per 3,000 indices. In clusters with many small indices (multi-tenant setups), this ceiling is often hit before the per-node shard count does.
Storage: match the tier to the access pattern
Not all data gets queried equally. Treating all data the same can lead to storage costs growing unexpectedly.
The Elasticsearch search speed guide states to use directly attached storage for hot data and leave at least half of system memory for the OS filesystem cache. Network-attached storage (NAS) adds latency to every read, and Elasticsearch performs many reads. Some NAS implementations also don’t correctly handle POSIX filesystem semantics, which can cause data corruption that only surfaces during recovery, not normal operation.
As data ages and gets queried less, ILM can help you move data to progressively cheaper storage:
Hot — Local SSD. High I/O, active indexing, full query performance.
Warm — HDD acceptable. No active indexing, lower query pressure.
Cold — Searchable snapshots. No replicas needed, roughly 50% cheaper than warm.
Frozen — Searchable snapshots. Up to 20x storage reduction vs warm, slower queries, Enterprise license.
The cold and frozen tiers use searchable snapshots: data lives in object storage (S3, GCS, or Azure Blob Storage) and Elasticsearch reads it directly. No replicas are needed because the snapshot itself is the source of truth. A benchmark on 90 TB of time-series data found that a hot-plus-frozen architecture cost roughly 8x less than keeping everything on hot storage.
To monitor disk usage per node, run:
GET _cat/allocation?h=node,node.role,disk.used,disk.avail,disk.percent&v
Here’s a working ILM policy that moves data through all four tiers:
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": { "max_primary_shard_size": "50gb" }
}
},
"warm": {
"min_age": "14d",
"actions": {
"shrink": { "number_of_shards": 1 }
}
},
"cold": {
"min_age": "30d",
"actions": {
"searchable_snapshot": {
"snapshot_repository": "my_repository"
}
}
},
"frozen": {
"min_age": "90d",
"actions": {
"searchable_snapshot": {
"snapshot_repository": "my_repository"
}
}
},
"delete": {
"min_age": "365d",
"actions": { "delete": {} }
}
}
}
}Your min_age values are the part worth thinking about. The right settings for your cluster depend on your actual query patterns, which almost never match what you assumed at design time. Check your logs before setting these. Which indices get queried daily? Which weekly? Which almost never?
On Elastic Cloud, you pick a hardware profile per tier and Elastic handles storage provisioning. While you don’t need to think about the hardware decision, you still need to think about which tier data ends up in.
What to watch for
None of these constraints fail loudly. You won’t get an error message telling you your shards are too small, your node is approaching its shard ceiling, or your hot tier is storing data that nobody hasqueried in months. What you will get, however, is slow queries, allocation warnings, and a storage bill that outpaces your data growth.
Check your shard sizes, check your shard count per node, and check which tier your data actually lives on. If you want to validate your configuration against real workload numbers, use Elastic’s load testing framework, Rally.
Frequently asked questions
How many shards can one Elasticsearch node handle? The soft ceiling is 1,000 per non-frozen data node, controlled by the cluster.max_shards_per_node setting. If you hit allocation failures and aren't sure why, that setting is the first thing to check. Raising it is possible but usually the wrong fix. Reducing primary shard count or widening your rollover window is more likely to address the root cause.
Is the “20 shards per GB of heap” rule still valid? No. This was deprecated in Elasticsearch 8.3. Keep shards between 10–50 GB, stay under 1,000 per node, and you’re covered.
When should I move data to cold or frozen tiers? Run GET _cat/indices?h=index,search.query_total&s=search.query_total:asc&v and look at the bottom of the list. Indices with near-zero queries are candidates for cold. Indices you're keeping for compliance or rare investigation belong in frozen. Both tiers use searchable snapshots and need no replicas.
Can I use network-attached storage for Elasticsearch? Not for the hot tier. NAS adds latency to every read, and some implementations mishandle POSIX filesystem semantics — causing corruption that only surfaces during recovery. Use directly attached SSDs for hot data. Warm and cold tiers are more tolerant of network storage since query pressure is lower.
How do I know if I’m rolling over too frequently? Check your shard sizes at rollover time with GET _cat/shards?h=index,store&v. If shards are consistently under 10 GB when the time trigger fires, your rollover interval is too aggressive. Widen it to weekly or monthly.
What if I’m not sure how much data my hardware can handle? Run Rally against your actual data and query patterns. Sizing guidelines give you a starting point and Rally can help you figure out specifics.
Sizing your first Elasticsearch cluster: what actually matters was originally published in codeburst on Medium, where people are continuing the conversation by highlighting and responding to this story.