The Register recently profiled an open-source keyboard where every key is its own module — magnetically attached, individually swappable, combinable into whatever board the builder wants. The designer says the sky's the theoretical limit on the size of the frankenboard you can build. It's a fun hardware story with an uncomfortable implication for anyone who carries an on-call phone: your alerting stack is the exact opposite of that design. It's a fixed layout, shipped in 2009, where every key does the same thing.
That's why a domain controller going offline and a print spooler restart on a receptionist's desktop produce the same artifact: an email to a distro list at 2 a.m.
If you run internal IT or work an MSP NOC, you know the numbers without being told. Your monitoring, RMM, and helpdesk each generate alerts independently. ConnectWise Automate fires a monitor failure, PRTG fires a sensor-down, Zabbix fires a trigger — all into the same shared inbox or Slack channel. Last night that was 200 notifications. Maybe 12 were real. Your on-call tech learned the noise pattern by week two and now swipes notifications away reflexively. Which is fine, until a real one — SQL Agent stopped, two hours before the finance batch job — gets the same swipe. You find out at 7:30 when the CFO calls.
Alert fatigue is not a volume problem. It's a signal quality problem. And the fix isn't "fewer checks" — it's alerts that carry their own context, and routing you assemble per service, per client, per hour of the day. Modular, like those keyboard bricks.
The Problem in Depth
Every check shouts at the same volume
Most alerting in the wild is flat. Nagios, aging PRTG and Zabbix deployments, and raw monitor output from RMM tools like Automate, NinjaOne, or Datto RMM treat a failed check as an event to transmit, not an incident to route. There is no severity policy deciding what wakes a human, no time-of-day awareness, and — for MSPs — no client context attached to the notification. A P1 database outage and a P4 informational event arrive with identical urgency: an email.
Your tools don't talk, so a human is the integration layer
The typical mid-size stack: monitoring in PRTG, Zabbix, or SolarWinds; RMM in Automate or NinjaOne; helpdesk in ConnectWise Manage, Autotask, or HaloPSA; paging via PagerDuty or Opsgenie. The "integration" between them is email forwarding and a technician doing copy-paste. The monitor detects the outage at 01:58. A human reads the email, creates the ticket, remotes in, starts diagnosing. That's 15–40 minutes of pure latency before real troubleshooting begins — and if the first the helpdesk hears of it is a user phone call at 8 a.m., your SLA report claims a 20-minute response while the ERP was actually down for six hours.
No deduplication: one root cause, 75 pages
A core switch reboots during a firmware update at a client site. In 90 seconds you get 60 endpoint-offline alerts from the RMM, six file-share alerts, four printer alerts, a UPS-on-battery event, and the site monitoring probe complaining about its own timeouts. Nothing correlates them. Your NOC tech starts working alert #3 (a laptop that was asleep anyway) while the actual root cause sits at position #71. One ISP blip becomes an hour of triage and a pile of tickets nobody needed.
Alerts with no context, so a human has to go look
"CPU high on SRV-APP-02" fires at 2 a.m. What was the value? 82% for four minutes, during the Veeam backup job — completely normal. But the alert carries no baseline, no trend, no duration, so a person logs in and checks. Multiply that by 30 alerts a night and you don't have monitoring; you have homework with a pager attached.
Maintenance windows nobody told the monitor about
Patch Tuesday rolls out via WSUS, servers reboot, and the monitors light up: agent unreachable, services restarting, disk checks failing mid-boot. In most tools, suppression is a per-check manual toggle, so nobody maintains it. The predictable consequence of planned change becomes overnight noise, and your techs mute the channel — which also mutes the unplanned stuff.
The business impact is measurable: inflated MTTR because the first 20–40 minutes is noise triage; SLA breaches the helpdesk report hides; NOC hours burned on non-events; and — the cost nobody budgets for — on-call burnout. Alert fatigue is a leading reason senior techs refuse rotations or leave altogether. When everything pages, nothing pages. The real 2 a.m. incident gets the same treatment as the print spooler.
How AlertMonitor Solves This
AlertMonitor was designed around one insight: every alert should be its own module — self-contained, context-complete, snapping into routing you compose per client, per severity, per schedule. Instead of one monolithic rule set, you build behavior modularly.
Context-rich alerts by default. Every alert carries the device, the client, what changed, and what healthy looks like. Not "Disk usage high on SRV-DB-01" but: "SRV-DB-01 D: at 91%, growing ~1.4%/day for three weeks, projected full in six days — SQL log backups are the likely contributor." The first version pages a human at 2 a.m. The second is a task someone handles at 10 a.m. Same underlying check, radically different outcome.
Smart deduplication and correlation. Repeated identical alerts collapse into a single alert with an occurrence counter, and dependent child alerts roll up under a root-cause parent. That switch firmware reboot produces one "site core down" event with 70 suppressed children attached — not 71 separate pages.
Multi-level on-call routing. Policies per client, per severity, per time-of-day, with ack-or-escalate chains: a P1 that is not acknowledged in 10 minutes escalates to tier 2, then to a voice call to the NOC lead. Nothing quietly dies in an inbox.
Maintenance window suppression. Schedule suppression windows per client or device group, tied directly to your patch deployments. Planned reboots generate zero pages, and a post-window health check confirms the environment actually came back clean. If something failed to restart properly, that is the alert you get — the only one.
One platform, zero copy-paste. Because monitoring, RMM, helpdesk, and patching live in one product, an alert auto-creates a ticket with device history and recent changes already attached, and remediation — restart a service, run a script, open a remote session — happens from the alert itself. No five-tool tab shuffle.
The old workflow: email lands → a human triages → remote in → diagnose → open a ticket in the helpdesk → update the RMM notes → document. The AlertMonitor workflow: a context-rich alert pages the right person → they acknowledge from their phone → one-click remediation → the ticket is auto-populated with the full timeline. Teams typically cut overnight pages by 60–80% once dedup and suppression are tuned — and the pages that remain are ones worth waking up for.
Practical Steps You Can Take Today
1. Baseline your noise. Pull the last 30 days of alerts and count them by source check. In almost every environment, fewer than 10% of checks generate 80% of pages. Fix those ten first.
2. Make your own checks context-rich. If you script custom checks, emit answers, not just failures. The classic 2 a.m. page is disk space — here is a sweep that reports actual numbers across your Windows servers:
$servers = Get-Content "C:/Reports/servers.txt"
Get-CimInstance -ComputerName $servers -ClassName Win32_LogicalDisk -Filter "DriveType=3" |
Select-Object @{n='Server';e={$_.PSComputerName}},
@{n='Drive';e={$_.DeviceID}},
@{n='FreeGB';e={[math]::Round($_.FreeSpace/1GB,1)}},
@{n='TotalGB';e={[math]::Round($_.Size/1GB,1)}},
@{n='PctFree';e={[math]::Round(($_.FreeSpace/$_.Size)*100,1)}} |
Where-Object { $_.PctFree -lt 15 } |
Export-Csv "C:/Reports/low-disk.csv" -NoTypeInformation
Free GB, total GB, percent free. Now an alert can say "4% free, shrinking 2% a week, act this week" instead of "disk high."
3. Wrap flaky services in a self-healing check with a paper trail. Restart automatically, but report what happened, so the alert that follows — if one follows — carries the full story:
$watch = @("MSSQLSERVER","Spooler","W32Time")
foreach ($name in $watch) {
$svc = Get-Service -Name $name -ErrorAction SilentlyContinue
if ($svc -and $svc.Status -ne 'Running') {
Start-Service -Name $name -ErrorAction SilentlyContinue
Start-Sleep -Seconds 5
$after = (Get-Service -Name $name).Status
Write-Output "SERVICE RECOVERY: $name on $env:COMPUTERNAME was $($svc.Status); restart attempted; now $after"
}
}
If it recovers, you get an informational event, not a page. If it does not, the page includes the recovery attempt, so the on-call tech starts already knowing something.
4. Do the same for the Linux side of the house:
#!/bin/bash
# Flag any local filesystem over 85% used
THRESHOLD=85
df -h --output=source,pcent,target | tail -n +2 | while read -r fs pct mount; do
usage="${pct//%/}"
if (( usage > THRESHOLD )); then
echo "DISK ALERT: $mount ($fs) is ${pct} used (threshold ${THRESHOLD}%)"
fi
done
5. Make escalation policy modular — per service, per client. This is the Lego-brick part: instead of one global notify-everyone rule, define composable policies. In AlertMonitor, a policy looks like this:
policy: sql-critical-contoso
applies_to:
client: "Contoso Manufacturing"
devices: ["SRV-SQL-01", "SRV-SQL-02"]
severity: P1
dedup_window: 15m
suppression:
maintenance_windows: ["patch-tuesday", "dr-test-q4"]
escalation:
- tier: 1 target: oncall-infra ack_within: 10m channels: [push, sms]
- tier: 2 target: noc-lead ack_within: 10m channels: [push, sms, voice] auto_ticket: create: true queue: "Infrastructure - P1" attach: [device_history, recent_changes]
Fifteen lines, and the SQL environment at one client now behaves completely differently from the print spooler at another — which is exactly how it should behave.
**6. Run a weekly tuning ritual.** Every page that should not have been a page gets fixed — a dedup window, a suppression rule, a baseline adjustment — not ignored. Teams that do this consistently for four weeks typically watch page volume fall off a cliff and stay there.
The keyboard hacker swapping magnetic modules to build a 200-key frankenboard has the right idea. Stop accepting an alerting layout someone else shipped in 2009. Build the one your on-call rotation actually needs — one module at a time.
## Related Resources
[AlertMonitor Alert Management & On-Call Operations](https://alertmonitor.ai/platform/alert-management)
[AlertMonitor Platform Overview](https://alertmonitor.ai/platform)
[Book a Demo](https://alertmonitor.ai/contact)
[Alert Management & On-Call Operations Resources](https://alertmonitor.ai/blog/alert-management)
Is your security operations ready?
Get a free SOC assessment or see how AlertMonitor cuts through alert noise with automated triage.