When deployments fail often enough, teams stop deploying. They batch changes, which makes each release larger, which makes failures more likely and harder to diagnose. It is a reinforcing loop and it is the real cost, well before you count the incidents.
Across a hundred-plus cluster estate we took that failure rate down by roughly eighty percent. Here is what actually contributed, in rough order of how much it mattered.
Environment parity, by a distance
The largest single cause of failed deployments was not the deployment. It was that staging had stopped predicting production, so a change validated in one behaved differently in the other.
Environments had been created at different times by different people, then adjusted independently. Nobody made a decision to let them diverge; it accumulated. Rebuilding all of them from the same code, so the only difference was the inputs, removed a whole class of failure that had previously been attributed to bad luck.
If staging does not predict production, every deployment is the first time you have run that configuration.
Declarative delivery with real reconciliation
Moving to GitOps mattered, but not for the reason usually given. The benefit was not that the desired state was in version control, though that helped reviews. It was that the deployed state was continuously compared against it.
Before, a cluster could drift from what anyone believed was deployed, and the discrepancy surfaced during the next incident. After, a divergence was corrected or surfaced immediately, which meant deployments started from a known state rather than an assumed one.
Rollback that is one step and always available
This does not reduce the failure rate. It reduces what a failure costs, which changes behaviour in a way that does reduce the rate.
When recovery means redeploying the previous version from scratch and hoping, teams deploy less often and in bigger batches. When rollback is one operation that always works, they deploy smaller changes more frequently, and smaller changes fail less. The second-order effect was larger than the first.
Health checks that mean something
A surprising share of apparent deployment successes were failures nobody had noticed yet, because readiness probes reported that a process was running rather than that a service could do its job.
Tightening probes so they exercised the dependency path made some deployments start failing that had previously passed. That looks like a regression on the dashboard and is the opposite: the failures were already happening, just later and further from the cause.
What contributed less than expected
More approval gates did nothing. Failures were not caused by insufficiently reviewed changes; they were caused by environments that could not be reasoned about. Adding review steps slowed delivery and left the rate roughly where it was.
Progressive delivery helped, but later than we assumed. Until parity and reconciliation were in place, a canary mostly told you that something was wrong without helping you understand why. It became genuinely useful once the ground underneath it was stable.
The order we would repeat
Parity first, because it removes the failures you cannot explain. Then reconciliation, so you always know what is actually running. Then cheap rollback, so teams start deploying smaller changes. Then progressive delivery, once the signal is trustworthy enough to act on automatically.
Done in the other order, each step is built on ground that moves, and the improvements do not compound.
