A Hundred Kubernetes Clusters: What Breaks That Does Not Break at Ten | The Software Geek
All postsKubernetes at Scale

A hundred clusters: what breaks that does not break at ten

3 min read

There is no wall you hit. That is what makes this difficult. Practices that are perfectly sensible across ten clusters degrade so gradually that by the time they are clearly broken, a lot has been built on top of them.

These are the ones that caught us, roughly in the order they became a problem.

Upgrades stop being events

With ten clusters, an upgrade is a project. Someone plans it, a few people are around for it, and it is done in a week. That model cannot survive a hundred clusters on a release cadence measured in months, because the work never finishes before the next version lands.

The shift is from upgrading clusters to operating a fleet that is continuously upgrading. That means canary clusters that take the new version first, a defined soak period, and automation that promotes through tiers without a human driving each step. It also means accepting that you will always have two or three versions in the estate at once, and your tooling has to tolerate that rather than assume uniformity.

Anything manual becomes a liability

A certificate renewed by hand once a year is fine. A hundred of them, on different schedules, owned by people who have since moved teams, is an outage waiting for a quiet week.

The general form of this: any task whose cost scales linearly with cluster count and which depends on a human remembering will fail. Not might. The interesting question at this scale is not whether something is automated but whether its failure is visible when the automation stops working.

Drift becomes the default state

Ten clusters can be kept identical by care and attention. A hundred cannot, because the rate at which small exceptions accumulate exceeds the rate at which anyone reconciles them by hand.

At this scale you do not prevent drift. You make it continuously visible and continuously corrected.

This is the argument for expressing cluster state declaratively and having something reconcile it constantly, rather than applying configuration and trusting it to stay. The value is not elegance. It is that the estate converges without anyone deciding to make it converge.

The control plane becomes a shared fate

Once every cluster depends on the same delivery mechanism, policy engine or registry, those become the highest-blast-radius components you operate. A bad policy that would have inconvenienced one team now blocks deployment everywhere at once.

The mitigations are unglamorous: stage changes to shared components the way you stage cluster upgrades, keep the policy set small enough that a human can reason about it, and make sure the delivery mechanism failing does not prevent you from fixing the delivery mechanism.

You stop being able to answer basic questions

Which clusters run version X. Which are running a workload that uses a deprecated API. Which have a policy exception nobody has revisited. At ten clusters someone knows. At a hundred, nobody knows, and finding out takes a day of scripting each time it is asked.

An inventory that answers those questions on demand is worth more at this scale than almost any individual optimisation, because it converts a day of investigation into a query. It is also the thing teams most consistently postpone, because it is not delivering a feature.

The habit that scales best

Whenever you do something to a cluster by hand, ask what happens when this is needed a hundred times. If the answer is that it would not get done, the manual action is not the fix, it is the symptom. Automating at fifteen clusters feels premature and is dramatically cheaper than automating at ninety.

Building something like this?

Tell us what you are working on and we will come back with a straight answer on whether we can help, and what it would take.

Schedule Free Consultation

More from the blog