The New Stack recently documented how WHOOP — the fitness wearable company — nearly drowned in vulnerability alerts: hundreds of thousands of findings per day. Their fix was not hiring an army of analysts, and it was not handing everything to a black box. It was an automated response workflow that triaged and routed the noise while keeping a human in charge of every decision that mattered.
Swap "vulnerability scanner" for "your monitoring platform, your RMM, and your helpdesk," and the story is identical for IT operations teams and MSPs. The average IT department is not short on alerts. It is short on signals. The people paying for that difference are your on-call technicians, your response times, and eventually your SLAs.
If you have ever muted your own monitoring tool's channel because it was "crying wolf" again, this post is for you.
Alert Fatigue Is Not a Volume Problem. It Is a Signal Quality Problem.
Here is the scenario every sysadmin recognizes. It is 2 a.m. Your phone buzzes. Alert #73 tonight: "Disk E: on FILE01 above 90%." You have seen 200 variations of this alert in the past month, and 198 of them were noise — a backup job spiking usage, a log volume that grows and shrinks, a check someone configured with a default threshold and never tuned. So you snooze it and go back to sleep.
Except this time it was real. The disk filled. The file shares went offline. The first person to notice was Karen in accounting at 8:15 a.m., via a helpdesk ticket that says "can't open the S: drive." By the time anyone connects the overnight alert to the morning ticket, you have lost six hours, restored from backup, and written a post-mortem that concludes, again, "we had the alert, we just didn't see it."
That is what alert fatigue actually costs: not annoyance, but outage time, SLA misses, and technicians who stop trusting their own tools.
The Problem in Depth: Why Your Stack Generates Noise Instead of Signal
Existing tools fire alerts. They were never built to be answered.
Standalone monitoring platforms (PRTG, SolarWinds, Zabbix, and their Nagios-era cousins) were architected around checks: one threshold, one device, one alert, one email. They have no concept of an incident lifecycle — nothing to acknowledge, nothing to deduplicate, nothing to escalate on a schedule. An "alert" is a line in an email inbox that twelve people share.
RMM platforms are not much better out of the box. Every offline endpoint, failed patch, and script error can generate a notification, and many MSPs leave the defaults on because turning them off feels risky. The vulnerability and patching side multiplies it: one missing Windows cumulative update can produce a finding per CVE per machine. Across 200 endpoints, a single missed patch Tuesday becomes thousands of discrete findings — exactly the dynamic WHOOP ran into with vulnerability scanning at scale.
Meanwhile, your helpdesk does not know any of it happened. The monitoring tool saw the disk fill at 2 a.m. The helpdesk found out at 8:15 from Karen. No system connected those two events, so a human had to — at the worst possible moment.
Why the gaps exist
- Siloed architecture. Monitoring, RMM, helpdesk, and patching were bought separately, deployed separately, and store data in incompatible models. Integration, where it exists, is a webhook or a CSV export.
- Alerts without state. Most legacy tools emit events, not alerts. An event has no owner, no acknowledgment, no snooze, no resolution, and no escalation if nobody touches it. It just sits in an inbox.
- Flat thresholds, no baselines. "CPU > 90%" means one thing on your SQL server during the backup window and another at 3 a.m. on a domain controller. Tools without baselines cannot tell the difference, so they page you for both.
- No maintenance awareness. Planned reboots, patch windows, and migrations trigger the same pages as real failures — which trains your team to dismiss pages.
- No client dimension for MSPs. Thirty clients' worth of alerts land in one undifferentiated stream. There is no per-client on-call routing, no per-client severity, no account-manager escalation path.
What it costs the business
- Mean time to acknowledge measured in hours, not minutes. The critical alert sits in a shared mailbox while the person responsible for it sleeps through it.
- Real incidents buried under noise. At a 200:1 noise-to-signal ratio, missing the disk-full alert is not a people failure — it is a design failure.
- SLA reporting that requires a spreadsheet merge. Response-time data lives in the monitoring tool; ticket data lives in the helpdesk. Nobody can reconcile them without manual work, so SLA reports are late, approximate, or simply never produced.
- Technician burnout and turnover. On-call engineers who get eight meaningless pages a night start sleeping through their phones — and then start looking for jobs where on-call means something. Replacing a good sysadmin costs far more than fixing your alerting.
How AlertMonitor Solves This
AlertMonitor was designed around a specific insight: alert fatigue is not a volume problem — it is a signal quality problem. You do not fix it by alerting less; you fix it by making every alert worth a human's attention.
Every alert carries full context. Not "Disk E: > 90%," but: device, client, what changed, and what healthy looks like. FILE01 disk E: crossed 90% — baseline for this server is 61% free, trend shows +14 GB over the last 6 hours, check stable for 90 days. A tech can triage that from a phone screen in five seconds. A bare threshold trip requires opening three consoles.
Smart deduplication. When a core switch flaps, most tools emit 61 alerts: one for the switch and sixty for the endpoints behind it that went "offline." AlertMonitor correlates and collapses them into a single incident with the root-cause device surfaced. A flapping check that fires repeatedly inside a short window is grouped with a counter — not re-paged every time.
Maintenance window suppression. Patch night generates zero pages. The activity is logged and visible, but the only thing that pages anyone is a deviation from the expected pattern — like FS01 not coming back after its scheduled reboot. Your team stops training itself to ignore notifications, because notifications only ever mean something.
Multi-level on-call routing. Escalation policies match operational reality: primary on-call via push and SMS immediately, secondary after 10 minutes without acknowledgment, IT manager after 25. Business-hours and after-hours paths differ. For MSPs, routing is per client and per severity — Acme Corp's critical server going down pages the Acme-designated tech, not the shared firehose.
Alert-to-ticket with no swivel-chair. A critical alert becomes a helpdesk ticket automatically, pre-populated with the device, check history, and context. The responding tech remotes into the machine with built-in RMM or pushes the patch that fixes it — same console, same timeline. When monitoring, helpdesk, RMM, and patching share one data model, the SLA report is a filter, not a two-hour spreadsheet exercise.
The human stays in charge. Like WHOOP's approach, automation handles the triage, correlation, and routing — the parts machines are good at. People make the decisions and do the fixing. No opaque auto-remediation silently rebooting production at 2 a.m.
The workflow, before and after
Old way: alert fires → email lands in shared mailbox → someone eventually sees it → someone checks which client and device → someone opens a ticket manually in the helpdesk → someone RDPs in with credentials from the password vault → the fix gets applied → the timeline exists across four tools and nowhere as one coherent report. Typical acknowledge time: 40+ minutes during business hours; overnight, until morning.
AlertMonitor way: alert fires with full context → on-call paged per policy in seconds → acknowledged from mobile → ticket auto-created and linked → remote session or patch push from the same pane → resolution logged, SLA captured automatically. Typical acknowledge time: under two minutes, around the clock. Teams consistently report overnight pages dropping from dozens to a meaningful handful — and every remaining page being one worth taking.
Practical Steps You Can Take This Week
1. Measure your noise before touching anything
Pull the last 30 days of alert history and rank your top 10 repeat offenders. If your current tool cannot produce that report in one click, that is itself a finding — you cannot fix what you cannot count.
2. Baseline what healthy looks like
Escalation only works when checks reflect reality. Two quick scripts to define "healthy" for your Windows fleet. Disk headroom across your critical servers:
$servers = "FS01","SQL01","DC01","APP01"
Get-CimInstance -ComputerName $servers -ClassName Win32_LogicalDisk -Filter "DriveType=3" |
Select-Object SystemName, DeviceID,
@{n='SizeGB';e={[math]::Round($_.Size/1GB,1)}},
@{n='FreeGB';e={[math]::Round($_.FreeSpace/1GB,1)}},
@{n='FreePct';e={[math]::Round($_.FreeSpace/$_.Size*100,1)}} |
Sort-Object FreePct | Format-Table -AutoSize
And auto-start services that are not running — the difference between "the box is up" and "the box is actually working":
Get-CimInstance -ClassName Win32_Service -Filter "StartMode='Auto' AND State!='Running'" |
Select-Object PSComputerName, Name, DisplayName, State |
Format-Table -AutoSize
Run these weekly for a month. The numbers you collect are your baselines — set alert thresholds against them, not against vendor defaults.
On Linux, a ten-second triage snapshot — disks over 85%, failed systemd units, and load:
#!/bin/bash
echo "=== Disks over 85% ==="
df -h --output=source,pcent,target | awk '$5+0 >= 85 {print "DISK:", $0}'
echo "=== Failed units ==="
systemctl --failed --no-legend | grep . || echo "None"
echo "=== Load ==="
uptime
3. Write an escalation policy that matches how your team actually works
In AlertMonitor, an escalation policy is explicit, versioned, and per client. The shape of it:
escalation_policy: critical-server-down
applies_to:
client: 'Acme Corp'
devices: ['FS01', 'SQL01', 'DC01']
severity: critical
dedup_window: 10m
suppress_during_maintenance: true
levels:
- after: 0m
notify: [primary-oncall]
channels: [mobile-push, sms]
- after: 10m
notify: [secondary-oncall]
channels: [mobile-push, sms]
- after: 25m
notify: [it-manager]
channels: [mobile-push, email]
auto_resolve_when: check_healthy_for_15m
Note what this policy does that a shared inbox cannot: it guarantees acknowledgment (escalation is automatic, not social), it respects maintenance windows, and it deduplicates a flapping check into one page instead of forty.
4. Put maintenance windows around every planned change
Patch night, reboot schedules, migrations — all of it goes into a window. Suppression is not ignorance: AlertMonitor still watches the device, still logs everything, and still pages if the pattern deviates — a server that does not come back, a service that does not restart. What stops is the predictable, worthless page.
5. Make every critical alert become a ticket — automatically
Policy, not preference: critical alerts auto-create helpdesk tickets. This is only cheap to do when monitoring and helpdesk share a platform, and it is the single change that makes your SLA reporting real instead of reconstructed after the fact.
6. Review the noise monthly and delete ruthlessly
Once a month, pull your ten noisiest checks. Tune the threshold, add a baseline, or delete the check. An alert nobody acts on is worse than no alert at all — it trains your team to stop reading the channel, and the one alert that matters becomes collateral damage.
Noise Is a Choice. So Is Signal.
WHOOP's story ends well because they did not try to solve hundreds of thousands of daily alerts by staring harder. They automated the triage and routing, and kept humans in charge of the judgment calls. That is exactly the design philosophy behind AlertMonitor's alert management: full context on every alert, deduplication and maintenance suppression that respect your team's sleep, and escalation policies that guarantee the right human sees the right signal fast.
Your on-call rotation does not need to be a hazing ritual. Point it at meaningful signals, and it becomes what it was always supposed to be: a fast, reliable path from problem to resolution.
Related Resources
AlertMonitor Alert Management & On-Call Operations AlertMonitor Platform Overview Book a Demo Alert Management & On-Call Operations Resources
Is your security operations ready?
Get a free SOC assessment or see how AlertMonitor cuts through alert noise with automated triage.