When an organisation reports resolution times in hours, the instinct is to assume the fixes are hard. Usually they are not. Break an incident into locating the fault, deciding what to do, and doing it, and the first phase is almost always the longest.
Across an estate spanning two cloud providers and more than a hundred clusters, that first phase was where nearly all of the elapsed time went. Here is what compressed it.
One place to start looking
Before, an engineer paged at three in the morning had to decide which tenant, which cloud and which console before they could begin. That decision was made with almost no information, and getting it wrong cost twenty minutes before they were back where they started.
A single entry point that spanned the estate removed that guess entirely. Not a dashboard showing everything at once, which is unreadable, but one starting place from which you could reach anything without first knowing where it lived.
Consistent labels made correlation possible
The reason cross-cutting incidents took hours was that the same service was labelled differently in different places, so nothing lined up. Correlating a spike in one system with errors in another meant a human recognising that two differently-named things were the same thing.
Correlation is a naming problem long before it is a tooling problem.
Agreeing one set of labels and enforcing it at collection did more for resolution time than any dashboard. Once metrics, logs and traces agreed on what a service was called, moving between them stopped requiring translation.
Alerts that carried context
An alert saying error rate high on a service is a notification that work is about to begin. An alert that arrives with the deployment that preceded it, the dependency that is also degraded, and a link to the relevant dashboard has done a chunk of the investigation before anyone opened a laptop.
This was mostly unglamorous work: attaching runbook links, including recent change events, and making sure the alert named the owning team rather than a generic channel.
Traces for the questions metrics cannot answer
Metrics tell you something is wrong and roughly where. In a distributed system with a deep call graph, they are poor at telling you which hop is responsible, and that gap is where a lot of the hours went.
Tracing on the critical paths, rather than everywhere, closed it. Full coverage is expensive and mostly unread. Coverage of the paths that generate incidents is affordable and gets used.
What did not help much
More dashboards. Every incident review produced a suggestion for a new one, and after a year there were hundreds, most unmaintained. During an incident nobody browsed them; they went to the two or three they trusted.
Lower alert thresholds. Tightening thresholds to catch problems earlier increased volume, and volume trained people to acknowledge without reading. That made resolution slower, not faster, because the signal was buried in things that resolved themselves.
The general lesson
If resolution time is measured in hours, instrument the phases before buying anything. Teams routinely invest in better fixing when the data says almost all the time went on looking, and those are different problems with different solutions.
