When alerting is bad, the complaint is usually about volume, and the proposed fix is usually thresholds. Tune the numbers, reduce the noise.
That is occasionally right. More often the alerts are firing on real conditions and arriving at people who cannot act on them, which produces the same experience as noise and is not fixed by tuning anything.
Route by ownership, not by system
The most common structural mistake is routing based on where an alert came from. Everything from the platform goes to the platform team, everything from a cluster goes to whoever runs clusters.
The result is a team receiving pages about workloads they did not write and cannot fix, whose only available action is to find the owner. That is a routing table implemented with a human in it.
If the first action on a page is to find who owns this, the routing is the incident.
Routing by service ownership requires a service catalogue that is accurate, which is one of the few genuinely load-bearing reasons to maintain one.
Alert on symptoms, diagnose with causes
Pages should fire on things users experience: requests failing, latency beyond what the service promises, work not being processed. Those are always worth waking someone for, and they stay meaningful as the implementation changes.
Cause-based alerts on CPU, memory or queue depth are valuable during diagnosis and poor as pages. High CPU with everything working is not an incident, and paging on it teaches people that pages do not necessarily mean anything.
Every page needs a decision attached
The test for whether something should page: if this fires at three in the morning, is there an action the recipient should take before business hours?
If yes, page, and put the action in the alert. If no, it is a ticket or a dashboard, and treating it as a page costs you attention you will need later. Most alerting systems accumulate a long tail of conditions that were worth knowing about once and have paged people at night ever since.
Carry context into the notification
An alert should arrive with what changed recently, what else is affected, which runbook applies and who else has been notified. Every one of those is a question the responder would otherwise ask in the first five minutes, at their least alert.
This is mundane work and it compounds. Alerts that carry their own context shorten every incident they are involved in, forever.
Review what fired and delete aggressively
A monthly look at what fired, how often, and what the responder did tells you most of what you need. An alert that fires regularly and is always acknowledged without action is worse than no alert, because it is training people to acknowledge without reading.
Deleting alerts feels risky and rarely is. If the underlying condition matters, it will show up again through a symptom-based page, which is the one you wanted in the first place.
