Not a bug. A trade-off you need to understand.
Update your profile picture, refresh the page on your phone, and for a second the old one is still there. Is that a bug? Most people's instinct is yes: I just saved this, why doesn't everything show it immediately?
It isn't a bug. It's eventual consistency, and if you build distributed systems, you're relying on it constantly, whether you've designed for it or not.
The Promise We Want, and What It Costs
The promise everyone wants is simple: you update something, and every service, every region, every cache sees it instantly. One version of the truth, always. That's strong consistency, and it's a completely reasonable thing to want.
It's also expensive. Guaranteeing that every node agrees before you consider a write done means coordination. And coordination across a network, especially across regions, means waiting on round trips you don't control. Do that on every write, at scale, and your system's latency is now bounded by your slowest node and your worst network hop.
So most distributed systems don't do that. They accept a weaker guarantee in exchange for speed and availability, and call it eventual consistency.
What Eventual Consistency Actually Means
Here's the mechanism: a write lands on one node (or a small subset of nodes). The other nodes find out later, through replication, gossip, or some other propagation mechanism, not as part of the original write. If no new updates come in, every node eventually converges on the same value.
That word "eventually" is doing a lot of work, and it's worth being precise about it. It's a promise about outcome, not about timing. Given enough time and no further writes, all replicas agree. In practice, "eventually" is usually milliseconds, sometimes seconds, and only occasionally something you'd notice. But the guarantee itself says nothing about how fast. It only says eventually, which is a liveness guarantee, not a timing one.
See the diagram on the site.
The CAP Trade-Off
This is where the CAP theorem shows up. It's less abstract than it sounds once you connect it to a decision you're already making. CAP says a distributed system can only guarantee two of these three properties at once, and in practice, partition tolerance isn't optional for any system that spans more than one node:
See the diagram on the site.
That leaves a real choice only once a partition actually happens: stay consistent, where every node sees the same data, or stay available, where every request gets a response, even if it's not the latest data. You can't fully have both during that partition.
Eric Brewer, who proposed CAP back in 2000, revised his own framing in 2012 for exactly this reason. Partition tolerance was never really a dial you could turn off for a system with more than one node, so treating it as an equal third option was misleading. The only decision that matters is what happens to consistency and availability once a partition actually hits.
Real systems announce which side of that decision they land on. CP (consistency + partition tolerance) systems, like MongoDB in its default configuration or HBase, choose consistency: rather than serve you something wrong, they'd rather reject the request. AP (availability + partition tolerance) systems, like Cassandra or DynamoDB, choose availability: they'll serve you data that might be a few writes behind rather than nothing at all, and lean on eventual consistency to close the gap afterward. CA (consistency + availability) systems, like a traditional PostgreSQL or MySQL install running in one place, get to keep both, but only because they were never asked to survive a partition between nodes. Spread that same database across two data centers, and CA quietly stops being on the table.
Eventual consistency is the deliberate choice of availability over strong consistency. You're saying: I'd rather my system keep responding, even with slightly stale data somewhere, than refuse to answer. It shouldn't have to wait until everyone agrees. For a lot of systems, that's exactly the right call. It's worth naming it as a call, not something that just happens to you.
Where You're Already Using It
This isn't exotic. You might assume eventual consistency is something only distributed-systems specialists deal with. Consider how much of your daily infrastructure already works this way.
DNS is the oldest example: update a record and it can take minutes to propagate everywhere, not milliseconds. CDNs cache content at edge nodes and take time to invalidate it after a change. Social media feeds (likes, comments, view counts) propagate asynchronously. Nobody expects a like count to update everywhere in the same instant. And if you run a primary/replica database setup, your read replicas are, by construction, eventually consistent with the primary. There's always some lag, even if it's usually small enough that nobody notices.
You've been relying on eventual consistency for years. The only question is whether you've been designing for it on purpose.
What This Means in Practice
Designing for eventual consistency on purpose looks different from tolerating it by accident. A few things that help:
- Show what the user submitted, not what's confirmed. This is optimistic UI. Update the screen right after the action. Don't wait on a round trip to every replica before showing the change.
- Retries should be idempotent. If an operation might have partially succeeded, retrying it again needs to be safe, not additive.
- Version your data. ETags, sequence numbers, or vector clocks let you detect conflicts instead of silently losing one write to another.
- Conflicts need to be resolvable, not just detectable. "Last write wins" is the easy default, but it isn't always the right one. Know when it's acceptable and when it isn't.
None of this has to be all-or-nothing at the system level, either. Most distributed databases let you dial consistency per operation instead of per deployment. Cassandra lets you ask for a QUORUM read on an account balance and an ONE read on an activity feed, in the same application. DynamoDB defaults to eventually consistent reads but gives you a strongly consistent read on request, at roughly double the read cost. MongoDB's writeConcern and readPreference settings do the same job. Pick the guarantee per query, not per database.
Know the specific failure patterns to watch for too. They're the same handful every time:
- Reading your own write and getting the stale value back (you just saved this, why doesn't it show?).
- Stale aggregates, where a count or total lags behind the records it's summarizing.
- Two users updating the same record concurrently, with one silently overwriting the other.
- Downstream actions (an email, a notification) firing off the back of data that's already out of date by the time they run.
None of these are exotic bugs. They're the predictable consequences of eventual consistency meeting code that was written as if it were strongly consistent.
Not every part of your system should take the eventually-consistent trade, either. Financial transactions (debits and credits) need to be atomic. Inventory reservation needs strong consistency, or two customers can both "successfully" claim the last item in stock. A revoked authentication token needs to be invalid immediately, not eventually. Eventual consistency is a sensible default for a lot of your system. It isn't a universal rule. Treat it as one, and you'll end up debugging a double-sold item at 2am.
Closing
Eventual consistency trades "always correct" for "always available." For most of what you build, that's the right trade. The mistake isn't choosing it. The mistake is not knowing you've chosen it.
The worst eventual consistency bugs I've seen didn't come from the trade-off itself. They came from nobody realizing they'd made a trade-off, until a user noticed their own change had disappeared.
PS: Let me know if I missed anything, ping me on Twitter/X or LinkedIn, and follow along there if you'd like more of this. Let's chat.
