A very old story, and still the best explanation of why on-call teams stop answering alerts. Read the story first. Then see how NGN helps IT teams anywhere put it right.
Once, on a hill above a village, a shepherd boy kept watch over the sheep. If a wolf came, he was to cry out, and the whole village would come running. The nights were long, and nothing ever happened.
One night, bored, he filled his lungs and shouted as loud as he could: Wolf! Wolf!
Down in the village, people dropped what they were doing, grabbed a torch and a pitchfork, and ran up the hill. Where is it?!
There was no wolf. Only the boy, laughing. Ha! Got you! The villagers walked back down the hill, grumbling.
A few nights later he did it again. And again. Every time they came running, and every time there was nothing there. Not again…
After a while, the villagers stopped running. When the shout came they shrugged, said It’s just the boy, and went back to their supper. In the pines, something was watching.
Then one night a wolf really did come out of the trees, low and quiet, straight for the flock. The boy shouted louder than he ever had: Wolf! It’s real this time!
Down in the village, everyone heard him. They had heard him a hundred times before. Nobody came.
By morning, the flock was gone.
Nobody believes a liar, even when he is telling the truth.
The moral, as Aesop told it more than 2,500 years ago.
Act II
Now tell it again, with a monitoring system
Swap the hill for a server room and nothing else needs to change. This is alert fatigue, and it plays out every night in IT teams everywhere.
The boy
NNagiosWarningDisk /var 82% on web-02
PrometheusFiringHighCPU: app-03 at 87%
Azure MonitorSev 3Swap in use on vm-db-02
Your monitoring
The boy is your monitoring
Nagios, Icinga, Prometheus, Zabbix, Datadog, AWS CloudWatch, Azure Monitor, Microsoft 365: every one of them will shout about everything if it is set up to. CPU at 87%, a certificate due in 60 days, a backup that ran six minutes long. Each sounds reasonable. Together they are noise.
The villagers
On call tonight
PPrimary47 pages since midnightMuted
SSecondaryPhone on silentAsleep
#noc ignore it, it’s the CPU one again
Your on-call team
The villagers are your on-call team
They came running at first. Then they learned that almost every alert needed nothing from them, muted the channel, and started treating alerts that way.
The wolf
IcingaCriticalBGP neighbour down on core-rtr-01
Unread0 acknowledged1 of 47 tonight
The real outage
The wolf is the real outage
The core router loses its internet connection at 3:07am. The alert arrives in the same channel, in the same format, as the forty-six false alarms around it.
The flock
OfficeOffline
OrdersFailing
PhonesDead
Staff idle140
Your business
The flock is your business
Customers who cannot order, staff who cannot work, phones that do not ring. That is what the wolf takes while nobody is looking, and nobody finds out until the first customer calls.
Spot the wolf
A quiet night in the alerts channel. Eight tools, each shouting in its own words, every line made to look urgent. One of them is the outage that will cost you tomorrow morning, and the engineer on call is reading this half asleep, on a phone.
That one. BGP neighbour down on core-rtr-01: the main site has lost its internet connection, and the alert looks just like the ten around it.
#alertsTuesday, 3:11 am · 11 new
PrometheusFiringHighCPU: app-03 at 87%
NNagiosWarningDisk /var 82% on web-02
DatadogWarnCertificate expires in 60 days: mail-01
IcingaWarningbackup-nightly ran 6 min long
ZZabbixProblemPing jitter 4 ms on syd-edge-1
Azure MonitorSev 3Swap in use on vm-db-02
IcingaCriticalBGP neighbour down on core-rtr-01
AWS CloudWatchAlarmcpu-high: app-03 at 91%
Microsoft 365AlertSign-in from a new device: admin
ZZabbixProblemNTP drift 40 ms on log-01
NNagiosWarningMail queue at 120 messages
The villagers did nothing wrong
When almost every alarm needs nothing from you, ignoring alarms is the rational response. Hospitals named the problem first: in 2013 The Joint Commission estimated that between 85 and 99 percent of alarm signals in hospitals do not require clinical intervention, and found alarm fatigue the most common contributing factor in alarm-related harm.1
IT is no different. Google’s Site Reliability Engineering book warns that the fatigue noisy alerts cause “can also cause serious alerts to be treated with less attention than necessary”.2Alert fatigue is not a discipline problem. It is a design problem, and it is fixed by giving the boy fewer reasons to shout.
Act III
A better ending
The fix is not a louder alarm or more people on call. It is a boy who only cries wolf when there is a wolf, and a village that knows exactly who goes when he does.
Wolf! Wolf!
Ha! Got you!
WOLF!
WOLF! It’s real this time!
Where is it?!
Very funny.
Not again…
Hold on, we’re coming!
It’s just the boy.
Ignore him.
Listen to the noise
We go through what actually fired over the last month: every alert, who it reached, and whether anyone needed to do anything.
Agree what a real wolf is
The business, not the engineers, names its emergencies: customers cannot log in, orders stop, the phones are down, data is at risk.
Watch for wolves, not rustling
Alerts fire on what users would notice (latency, traffic, errors and saturation, Google’s four golden signals) and on how fast a service is burning through its error budget, not on every CPU spike.2,3
Send each cry to the right person
Related alerts are grouped into one notification, knock-on alerts are suppressed while their cause is firing, and each alert is routed by service and severity to the person who owns it, escalating until someone acknowledges it.4,5
One cry, one plan
Every alert that wakes someone links to a short runbook: what it means, who is affected, what to check first and who to call.
Train the village, and keep it honest
Your engineers learn to write alerts and run the weekly fifteen-minute review: what fired, did anyone need to act, does it stay. Alerts that never need action are deleted.
The same wolf, cried properly
The difference is not a louder alarm. It is an alert that says what is at stake, who is affected and who is already on it, and quietly folds the noise away.
Before
IcingaCriticalBGP neighbour down on core-rtr-01
+ 46 more alerts like it tonight. Which one matters? Nobody can tell from this.
After
P1Sydney office is offlineAcknowledged 03:09
Sites
Sydney head office, Parramatta warehouse
Services
Internet, phones, VPN, online orders
Users affected
140 staff and every customer order
Likely cause
core-rtr-01 lost its BGP session to the upstream at 03:07
38 related alerts grouped into this one12 knock-on alerts held back
Detected from what users see, not CPU
Paged Network on-call: call, SMS, Teams
Acknowledged, runbook opened
Would have escalated to the secondary
Runbook: Upstream BGP session down
≤ 2incidents per 12-hour on-call shift, the most Google’s SRE teams allow
1 : 1alerts to real incidents: the ratio Google’s SRE book tells teams to tune towards
100%of pages actionable: “Every page should be actionable”
How we work with your team
NGN works with IT teams in Australia and around the world, on site or remotely, with the tools you already run.
TrainHands-on sessions for your on-call engineers: writing alerts that earn a page, runbooks, routing and the weekly review.
TuneWe rebuild the alert set you already have, in Icinga2, Prometheus, Grafana or whatever you run, so what is left is trusted.
ImplementNew monitoring and alert routing, built and documented: Prometheus and Alertmanager, Icinga2 and Grafana, notifying through PagerDuty, Microsoft Teams, Slack, email, SMS or a phone call.
AnswerOr hand us the pager: NGN monitors around the clock, and our engineers take the call and act on it.
What does the boy who cried wolf have to do with IT monitoring?
It is the same story. A monitoring system that raises alarms about everything teaches the people on call that alarms do not mean anything, so when a real outage arrives it is ignored like the false alarms before it. In IT this is called alert fatigue, and like the fable, the fault lies with the alarms, not with the people who stopped answering them.
What are the warning signs of alert fatigue in an IT team?
An alerts channel or email folder nobody reads, alerts that are routinely acknowledged without being looked at, the same alert firing every night, outages that customers report before monitoring does, and one experienced engineer who just knows which alerts are real. Any one of these means the alert set needs rebuilding.
Will cutting the number of alerts mean we miss real problems?
Not if it is done properly. The alerts that remain are built around what customers would notice, so they catch real problems more reliably than a wall of CPU and disk warnings did. We also track incidents that arrived with no alert, which shows straight away if anything was cut too deep.
How long does it take to fix a noisy monitoring system?
Usually a couple of weeks for the rebuild itself, using the tools you already have, such as Icinga2, Prometheus or Grafana. The weekly review that keeps it quiet afterwards takes about fifteen minutes.
What standards does NGN follow when it rebuilds alerting?
The practices Google published for its own Site Reliability Engineering teams: every page must be actionable, alerts fire on symptoms users would notice rather than internal causes, paging is driven by how fast a service is using up its error budget, and an on-call engineer should face no more than two incidents in a twelve-hour shift. Alerts are grouped, suppressed and routed in tools such as Prometheus Alertmanager, and escalate until someone acknowledges them.
Can NGN be the on-call team that answers our alerts?
Yes. NGN monitors around the clock, with alerts routed by severity to our engineers and the escalation path you choose outside business hours, so a real alarm at 3am reaches someone who acts on it.
We have used Next Generation Networks for several custom solution projects that went well beyond the standard scope listed on their website. Every time, the team has been exceptional—genuinely enthusiastic about tackling complex challenges and consistently delivering solutions that exceed…
Tell us about your alerts. We rebuild noisy monitoring into something a team will answer, usually in a couple of weeks, with the tools you already have.
