This is Part 7 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.
Last time, we ended on a question that sounded simple but isn't.
Aisha updated her profile picture. Her new photo is now saved in the database. But the cache is still holding onto the old one, completely unaware that anything changed. So every request for Aisha's profile gets served the old data. Confidently. Instantly. Incorrectly.
How does a cache know when the data it's holding is no longer correct?
Think about what we've actually built at this point. We have an application that responds fast, scales horizontally, and avoids hammering the database with repeated identical queries. From a performance standpoint, it looks great.
But Aisha's friends are loading her profile and seeing a photo she replaced five minutes ago. The system isn't slow anymore. It's wrong.
Speed and correctness are two different things. We optimized hard for one, and quietly broke the other.
Engineers have a name for this problem: cache invalidation. It refers to the challenge of keeping the data in your cache consistent with the data in your database, as that underlying data changes over time.
It turns out to be one of the genuinely hard problems in building software systems. Not hard in a complicated-algorithm way. Hard in the way that every solution has a catch, and the right answer always depends on what you're willing to accept.
Let's think through it together.
--
Section 1: When Cached Data Lies
It's worth sitting with the problem a little longer before rushing to fix it, because the damage stale data can cause varies enormously depending on what's being cached.
Consider a few examples.
Your application caches the list of trending articles. An hour later, the list has changed. New articles have risen, old ones have faded. But the cache still serves the original list. Users see slightly outdated trending content. That's mildly annoying, but nobody gets hurt.
Now imagine your application caches a product's price. A flash sale begins and the price drops by 40%, but the cache still confidently serves the old price. Users add the item to their cart expecting a discount they aren't going to get. That's a real problem: frustrated users, support tickets, potential refund requests.
Now imagine your application caches a user's account permissions. A user is suspended by an admin, but the cache still serves the old permissions. That suspended user continues accessing parts of the system they should be locked out of. That's a security issue.
The underlying problem is the same in all three cases. The cache is serving data that no longer matches what's in the database. But the consequences range from "mildly stale" to "genuinely dangerous."
This is the tax that caching collects. The cache made your system faster, but it did so by creating a second place where data lives. And here is a rule worth remembering, because it applies far beyond caching:
Any time data lives in two places, those two places can disagree.
The question is never whether they will disagree. Given enough time and enough writes, they will. The question is always: how long are you willing to let them disagree, and what happens to your users when they do?
There's a famous saying among engineers, sometimes credited to Phil Karlton:
"There are only two hard things in computer science: cache invalidation and naming things."
It's been repeated enough to become a cliché. But the reason it's stuck around is that it's true. Let's see why.
--
Section 2: The First Instinct: Delete It When It Changes
The most obvious approach is also the most direct one. If the cached data might be wrong because the underlying data changed, then when you change the underlying data, just remove the cached version.
The logic is simple: an absent cache entry is honest. It says "I don't know." A stale cache entry lies. So if something changes, delete the cached copy, and let the next request go back to the database for a fresh answer.
This pattern is called Cache Aside, and it works like this:
READ path:
1. Check the cache.
2. If hit → return cached data.
3. If miss → fetch from database, store in cache, return data.
WRITE path:
- Write the updated data to the database.
- Delete the corresponding entry from the cache.
Let's trace through Aisha's profile update.
Aisha updates her profile picture:
App Server → Database: "Update profile for aisha"
App Server → Cache: "Delete profile:aisha"
Next request for Aisha's profile:
App Server → Cache: "Do you have profile:aisha?"
Cache: "No." → Cache Miss
App Server → Database: "Get profile for aisha"
Database returns the new, updated profile.
App Server → Cache: "Store this as profile:aisha"
App Server → User: here's the updated profile
The next person who visits Aisha's profile gets a cache miss, goes to the database, and the cache gets repopulated with the correct, up-to-date data. From that point on, subsequent requests are hits again.
It's clean. It's honest. And for many situations, it works well.
But notice what this approach requires: the application has to know, every single time it writes data, which cache entries to delete.
For something like a user profile, that's manageable. Write to the user table, delete the profile:aisha key. One write, one delete.
But systems get complicated. Imagine a product page that displays the product name, its current price, its average rating, and a list of the five most recent reviews. That data might come from four different database tables. It might be composed by three different parts of your application. When a new review gets posted, which cache keys need to be deleted? When a price is updated, what about any cached search results that also displayed that price? When a product is renamed, does the cache for every page that ever mentioned it need to be cleared?
Suddenly "just delete it when it changes" requires an exhaustive mental map of every cache entry that depends on every piece of data. Miss one entry, and the cache serves a lie.
The harder the system, the harder it is to maintain that map without mistakes.
--
Section 3: Letting Time Do the Work
Looking at the mess that Cache Aside can become in complex systems, you start wishing the cache would just clean itself up automatically, without requiring the application to track every dependency.
And there is a way to do exactly that. Instead of actively deleting cache entries when data changes, you give each cache entry a lifespan. After that lifespan expires, the cache entry is gone, regardless of whether anyone told it to leave.
This is called a TTL, which stands for Time-To-Live.
When you store something in the cache, you attach an expiration time to it. The cache holds onto it for that duration, serves it freely to whoever asks, and then discards it automatically when the time is up. The next request after expiration becomes a cache miss and fetches fresh data from the database.
SET trending_articles "[...data...]" WITH TTL = 60 seconds
→ For the next 60 seconds: cache hits, database never touched.
→ After 60 seconds: entry expires, next request is a miss.
→ Fresh data is fetched from database and cached again for another 60 seconds.
The beauty of TTL is that the application no longer needs to track which cache entries to invalidate. It just sets a reasonable expiration and lets time handle it. Every piece of cached data has a built-in expiry date. Nothing lives forever.
For data that changes infrequently and where brief staleness is acceptable, TTL is elegant. Trending articles that get recalculated every few minutes? Cache them with a 60-second TTL and stop worrying about it. Homepage banners that a marketing team updates once a day? A 5-minute TTL works fine.
But TTL introduces its own uncomfortable question.
How long should a cache entry live?
That question sounds like a detail, but it's actually where a real tension lives.
Set the TTL too short, and you're sending requests to the database constantly. The cache barely helps because entries expire before they have a chance to absorb much traffic. You're essentially paying the overhead of maintaining a cache without getting much of the benefit.
Set the TTL too long, and you risk serving stale data for an extended window. If a product price is cached with a 24-hour TTL and a flash sale starts, users could be seeing the wrong price for hours before the cache naturally expires and corrects itself.
There's no universal right answer. A TTL is always a tradeoff between how fresh the data needs to be and how much database load you're willing to accept. You pick a number that balances those two concerns for your specific use case, knowing that you can't fully optimize for both at once.
And here's the deeper issue. Even with a well-chosen TTL, there's still a window where the cache is wrong. Maybe just 30 seconds. Maybe just 5. But in those 30 seconds, thousands of users could be served stale data.
For some applications, a 30-second window of stale trending articles is completely fine. For a stock trading platform serving real-time prices, 30 seconds of stale data is a disaster.
The right TTL is the one that fits what your users actually need, not the one that happens to be technically convenient.
--
Section 4: Keeping Cache and Database in Step
Cache Aside requires the application to manually delete entries on every write. TTL lets entries expire on a schedule but tolerates a window of staleness. Both involve some moment in time where the cache and the database disagree.
What if the requirement is stricter than that? What if the cache must never serve data that doesn't match the database?
That's where Write-Through caching comes in.
The idea is a small but significant shift in how writes work. Instead of writing to the database and then figuring out what to do with the cache separately, the application updates both at the same time, in the same operation.
WRITE path with Write-Through:
1. Write the updated data to the database.
2. Write the updated data to the cache.
3. (Done. Cache and database now agree.)
Let's trace through Aisha again.
Aisha updates her profile picture:
App Server → Database: "Update profile for aisha" ✓
App Server → Cache: "Update profile:aisha" ✓
Next request for Aisha's profile:
App Server → Cache: "Do you have profile:aisha?"
Cache: "Yes, here it is." → Cache Hit (with the correct new data)
No staleness. No window of disagreement. The cache is updated at the same moment as the database, so every read after a write immediately sees the correct data.
This sounds ideal, and in some ways it is. Applications where incorrect data is genuinely costly (financial balances, inventory counts, access permissions) often lean toward Write-Through for exactly this reason. The cache and database stay synchronized by construction.
But Write-Through has its own cost, and it's worth being honest about it.
Every single write now has to update two places instead of one. That adds some latency to write operations. More importantly, it means the cache now contains entries for things that might almost never be read. When you write-through, you're pre-populating the cache on every write, regardless of whether anyone is going to read that data soon. You're paying the cost of caching data that might sit there unused.
Compare that to Cache Aside, where the cache only fills up with data that someone actually asked for. Cache Aside is demand-driven: things enter the cache because they were requested. Write-Through is write-driven: things enter the cache because they were modified, whether or not anyone will read them next.
Neither is wrong. They have different shapes.
Cache Aside:
Reads are slightly slower on first miss.
Writes are fast (just the database).
Cache contains what people read.
Risk: stale data between write and next read.
Write-Through:
Reads are always fast (cache is always up to date).
Writes are slightly slower (two destinations).
Cache may contain things nobody ever reads.
Risk: wasted cache space on infrequently-read data.
And even Write-Through isn't a complete solution. What happens if the write to the database succeeds but the write to the cache fails? Or the write to the cache succeeds but something goes wrong before the database write commits? Now the two places are out of sync again, through failure rather than design.
Systems where data lives in more than one place have to reason about failure. Always. That's a topic big enough for its own conversation, but it's worth flagging now: even a well-designed Write-Through strategy requires thinking about what happens when individual steps fail.
--
Section 5: There Is No Perfect Answer
By now, you've seen three approaches to cache invalidation. Each one appeared because the previous one had a flaw:
Cache Aside was honest and simple, but required the application to track every cache dependency. Miss one, and you serve stale data. Scale the system, and tracking those dependencies becomes a maintenance burden.
TTL removed that burden by letting time handle invalidation automatically, but introduced a guaranteed window of staleness, and forced an uncomfortable question about how long that window should be.
Write-Through eliminated the staleness window by synchronizing writes, but slowed down write operations and populated the cache with data that might never be read.
There is no fourth option that fixes all three problems at once. Every approach is a different answer to the same underlying tension: the cache exists to avoid work, but avoiding work means tolerating some risk that what you're serving isn't perfectly current.
The choice between these strategies isn't about which one is correct. It's about which tradeoffs you can live with, given your specific application.
A social media platform caching post counts can tolerate a few seconds of staleness. If a post shows 1,042 likes instead of 1,043, no one is harmed. A payment system caching a user's account balance cannot tolerate the same thing. If the balance is wrong even for a second, someone could be overcharged or allowed to spend money they don't have.
The same engineer, working on both systems, would make different choices. Not because one engineer is more experienced than the other, but because the data is different, the consequences are different, and therefore the acceptable tradeoffs are different.
This is what makes cache invalidation hard. Not the implementations themselves, which are learnable in an afternoon. What's hard is developing the judgment to know which approach fits which situation, and being honest about what you're giving up either way.
Phil Karlton was right. It really is one of the hard problems.
--
Conclusion
Let's trace the path of this article.
We started where Part 6 left us: Aisha's profile picture had been updated in the database, but the cache was still confidently serving the old one. That was the problem. Not that caching is broken, but that a cache holding onto old data isn't just slow to update. It's actively wrong.
We looked at three ways engineers have learned to handle this.
Cache Aside keeps things simple by deleting cache entries when the underlying data changes, trusting the next read to fetch fresh data and repopulate. It works, but it puts the burden on the application to know what to delete, a burden that grows with the system's complexity.
TTL offloads that burden to time. Set an expiration, let it self-clean. The tradeoff is accepting that stale data will exist for the duration of that window, and that choosing the right window is a judgment call with no universally correct answer.
Write-Through keeps the cache and database synchronized on every write, so reads are always accurate. The tradeoff is slower writes and a cache that may fill with data nobody ever requests.
Every one of these strategies is in active use in production systems today. Often, the same system uses all three, with different strategies applied to different types of data depending on how fresh that data needs to be.
Now step back and look at how far this series has come.
We started with a single server handling a single user. We added more servers when one wasn't enough. We added a load balancer to distribute the traffic across them. We added a cache to stop the database from answering the same question thousands of times. And in this article, we learned how to keep that cache honest as data changes underneath it.
At each step, we made the system handle more load. And at each step, a new problem appeared just past the solution we'd just built.
That pattern continues.
Even with a well-designed cache, there are requests that can never be served from one. Personalized content. Real-time transaction histories. Queries that are unique to each user and can't be pre-stored. Every one of those requests goes straight to the database, every single time.
As the application grows from thousands to millions of users, that one database starts receiving millions of different questions simultaneously. No amount of caching can absorb that. The database itself becomes the ceiling.
So what do you do when the database is the bottleneck, caching can't help, and there's simply more traffic than one machine can handle?
That's where we're headed in Part 8.