If your monitoring wakes you at 3am, it is not monitoring

Every monitoring system we inherit has the same shape. Thousands of checks. Hundreds of alerts a day. A dedicated email folder nobody opens. And an engineer who can tell you, without looking, which three alerts actually mean something.

That engineer is the monitoring system. The software is just generating noise around them.

The symptom everyone recognises

Alert fatigue gets framed as a people problem: the team has become complacent, they need to take alerts more seriously. That framing is wrong and it makes the situation worse.

If 95% of alerts require no action, ignoring alerts is the correct strategy. The team has correctly learned the base rate. You cannot train that away, and you should not want to: an engineer who reacts with full urgency to every one of two hundred daily notifications will be useless within a month.

Alert fatigue is not a discipline problem. It is a design problem, and it is fixed by deleting alerts.

The dangerous consequence is not the noise itself. It is that the real alert, the one at 3am that genuinely matters, arrives into a channel everyone has learned to ignore, and it looks exactly like the other forty that night.

One question that fixes most of it

For every alert that wakes a human, ask:

Applied honestly, this typically removes 80 to 90% of what a legacy monitoring system pages on. The resistance it meets is always the same: "but we want to know about it."

Wanting to know is legitimate. It is just not the same as needing to be interrupted. Those are three different destinations, a page, a ticket queue or a dashboard, and the core mistake in most noisy systems is that everything was routed to the first one because it was the default.

Alert on symptoms, not causes

The second change is what you alert on at all.

Most inherited systems alert on causes: CPU above 90%, memory above 80%, a process restarted, a queue over a thousand messages. Each sounds reasonable. Collectively they generate enormous noise, because none of them reliably means anything is wrong.

High CPU might be a batch job doing exactly its job. A restarted process might be a routine deployment. A thousand queued messages might be normal for a Tuesday.

Symptom-based alerting asks instead: what would a user notice?

The right column has three properties the left lacks. It is far less frequent. It almost always matters. And it stays correct when the system underneath changes: you can replace the entire backend and "requests are erroring" is still the right alert, while every CPU threshold needs revisiting.

Note the queue example in particular. Depth is meaningless without throughput; a queue of ten thousand draining fast is healthy, a queue of fifty that has not moved in an hour is broken. Age captures what you actually care about.

How to rebuild a noisy system

The instinct is to tune thresholds one at a time. That takes a year and leaves the underlying design intact. We start from the other direction.

1. Stop everything from paging. Route all existing alerts to a low-priority channel. This feels reckless. It is not, because nobody was reading them anyway, and you have just made that explicit instead of implicit.

2. Write down what would constitute an emergency. Ask the business, not the engineers. The answers are usually short and concrete: customers cannot log in, orders are not processing, the phones are down, data is at risk. Most businesses produce five to ten.

3. Build an alert for each one, from the user's perspective. Not "is the database up" but "can an order complete". Synthetic transactions are excellent here: a script that actually logs in and performs the critical path every minute tells you more than fifty infrastructure checks.

4. Page on only those. Everything else goes to tickets or dashboards.

5. Add back only what earns it. After each incident, ask whether an alert would have caught it sooner. If yes, add exactly that one. The alert set grows from evidence rather than anxiety.

Most organisations land somewhere between eight and twenty paging alerts for an environment that previously had several hundred. Coverage does not get worse. It gets better, because the remaining alerts are read.

Every alert needs a runbook

An alert that fires at 3am and says DiskSpaceWarning: prod-db-02 is a puzzle, not an instruction. The engineer has to work out what it means, whether it matters, and what to do, while half asleep.

Every paging alert should link to a short document answering four questions:

• What does this actually mean? In plain language.
• What is the user impact? Is anyone affected right now?
• What do I check first? The two or three most likely causes, in order.
• What do I do about it, and who do I escalate to?

This has a useful side effect. If you cannot write a runbook for an alert, if the honest answer to "what do I do about it" is "have a look, I suppose", then that alert has failed the test in the first section and should not be paging anyone. The runbook requirement is a filter as much as it is documentation.

Measuring whether it worked

Track three numbers, monthly:

• Pages per on-call shift. A common benchmark is no more than about two on average. Above that, attention degrades and response quality falls with it.
• Actionable rate. What proportion of pages resulted in someone doing something? Below about 75%, you still have noise. This is the single most useful number.
• Incidents with no alert. Problems that a customer reported before monitoring did. This is the check on over-correction: if it is climbing, you cut too deep.

Then hold a fifteen-minute review each week: what fired, was it actionable, and does it stay? Alerts that fire repeatedly without action get deleted or downgraded. This meeting is what stops the system drifting back, because noisy monitoring is not something that happens once. It accumulates, one reasonable-seeming check at a time.

The goal is simple to state and surprisingly rare in practice: when a page arrives, everyone believes it. That is worth more than any amount of coverage nobody reads.