Google Search is arguably the most heavily instrumented product on the planet. It has unlimited engineering talent, more telemetry than most national censuses, and a direct line to every AIOps vendor's dream customer: itself. And yet The Register's headline this week says it all — Google's search revenue Ferrari hit an AI-powered speed bump, with a subhead dripping sarcasm: just wait 'til we add more AI and this thing'll really fly.
There is a lesson in there for anyone who runs a NOC, an on-call rotation, or a helpdesk at 2 AM, and it is not flattering to our industry.
Every monitoring and RMM vendor has spent the last two years bolting "AI" onto their alerting. Anomaly detection. Predictive dashboards. An AI copilot that summarizes incidents you never wanted in the first place. Meanwhile, the actual experience of the on-call technician has barely moved: the 2:14 AM page for a disk that crossed 85% for six minutes, the branch office outage you learned about from the office manager's Slack message, the forty minutes of triage across four consoles before you even know if the incident is real.
The problem was never a shortage of intelligence. The problem is that the signal reaching your phone at night is garbage — and adding AI to garbage produces smarter-sounding garbage.
The Problem in Depth: Your Alert Pipeline Optimizes for Volume, Not Signal
Static thresholds generate lies. A disk alert that fires at 85% cannot tell the difference between a 2TB backup volume that has sat at 90% by design for a year and a 200GB OS volume filling at 2GB per day that will be dead in a week. Same threshold, same page, wildly different urgency. The tool doesn't know what healthy looks like for each device — it only knows a number somebody typed into a settings dialog in 2019.
Cascading failures bury the root cause. A core switch drops and your monitoring tool faithfully reports 150 endpoints offline, 30 servers unresponsive, and 12 services down — 400 alerts in 90 seconds. The one alert that matters, the switch, is number 312 in the queue. Every experienced tech responds by skimming, and skimming is exactly how a real, unrelated incident gets missed inside the noise.
No deduplication means flapping equals flooding. One interface that bounces every ten minutes produces six pages an hour. Over a weekend, that is 72 pages about the same thing. By page ten, the on-call tech has muted the channel. That is how you end up in the post-incident review explaining why "nobody saw the alerts."
Zero context turns every alert into an investigation. "CPU high on APP-03" tells you nothing. Running what process? Since when? Did something change, or has this box always run hot? Answering that means RDP, Task Manager, Event Viewer, and the change log that lives in someone's spreadsheet. Fifteen minutes of triage before you even know whether to care.
Maintenance windows don't exist, so patch night manufactures fiction. You push Windows Server updates to 60 machines on Saturday night. Sunday morning there are 400 alerts, every one self-inflicted. The team learns to ignore weekend alerts — which is precisely when the storage array decides to fail.
The tools don't talk to each other. Monitoring says the service is down. The RMM could restart it. The helpdesk needs a ticket with an SLA clock attached. Because they are separate products, a human swivel-chairs between consoles, copy-pasting device names and typing the same context three times. The SLA report gets assembled by hand in Excel because monitoring data and ticket data live in different universes.
And here is where the AI-washing comes in. When vendors saw "alert fatigue" trending, many shipped black-box anomaly scoring instead of fixing the fundamentals. Now instead of 400 clear alerts you get 40 cryptic "anomalous behavior clusters" you cannot tune, cannot explain to a client, and cannot map to an SLA. The Ferrari got a new spoiler. It still does not corner.
The operational results are the ones every practitioner recognizes: a large share of pages that are duplicates or pure noise, mean-time-to-acknowledge creeping upward as techs triage by ignoring, MTTR inflated by investigation overhead, and on-call rotations that burn out the people you can least afford to lose.
How AlertMonitor Solves It: Signal Quality First, Not AI Sprinkles
AlertMonitor was designed around one insight: alert fatigue is not a volume problem — it is a signal quality problem. Fix the signal, and the fatigue takes care of itself.
Every alert carries full context. Device, client, what changed, and what healthy looks like for that specific machine — not a generic threshold. The alert for the backup volume does not say "disk 90%"; it says "this volume normally sits at 90%, what is notable is the 40GB jump in the last hour." You evaluate the incident from the alert itself instead of after three RDP sessions.
Smart deduplication collapses storms. The flapping interface becomes one incident with a counter, not 72 pages. The switch failure becomes one root-cause incident with every affected device attached, so the first page you receive includes the blast radius.
Maintenance window suppression tied to what you actually do. Because patch management lives in the same platform, patching a group of servers can automatically suppress their alerts for the duration of the job. Patch night stops generating noise, and a genuine fault during a window still gets through on the devices not being patched.
Escalation policies that behave like a real rotation. Multi-level, configurable on-call routing: page the primary, escalate to the secondary after 10 minutes unacknowledged, reach the NOC manager at 25. Scoped per client, per severity, per time of day — critical for MSPs juggling a dozen client environments from one NOC. An email to a distribution list is not an escalation policy, and AlertMonitor does not pretend it is.
The workflow, side by side.
Old way: alert email → open monitoring console → open RMM → RDP in → check event logs → manually create a helpdesk ticket → paste context into it → remediate → update the ticket → write it up. Twenty to forty minutes of triage before resolution even starts.
AlertMonitor way: the alert arrives with device, client, change context, and health baseline → a ticket is created automatically with that context attached → acknowledge → remediate directly from the alert (restart the service, run the script, push the patch) → the ticket updates itself. Two to five minutes for the same class of incident.
Because monitoring, RMM, helpdesk, and patching share one data model, there is no swivel-chair and no end-of-month reconciliation. Your SLA report is a query, not a spreadsheet art project.
Practical Steps You Can Take This Week
1. Quantify your noise before you argue about it. Export your alert history to CSV and count the repeats:
# Find your noisiest alerts. Adjust Device/AlertName to match your export's columns.
Import-Csv .\alert-export.csv |
Group-Object Device, AlertName |
Sort-Object Count -Descending |
Select-Object -First 15 Count, Name |
Format-Table -AutoSize
If the top row shows a count of 60 or more, you have found your flapping offender. One deduplication rule for that alert will delete most of your overnight pages.
2. Replace static disk thresholds with rate-of-change awareness. A single snapshot does not tell you urgency; growth rate does. Sweep your Windows servers and list every volume that is both low and small enough to matter:
$servers = Get-Content .\servers.txt
Get-CimInstance -ComputerName $servers -ClassName Win32_LogicalDisk -Filter "DriveType=3" |
Select-Object SystemName, DeviceID,
@{n='FreeGB'; e={[math]::Round($_.FreeSpace/1GB,1)}},
@{n='SizeGB'; e={[math]::Round($_.Size/1GB,1)}},
@{n='FreePct'; e={[math]::Round($_.FreeSpace/$_.Size*100,1)}} |
Where-Object FreePct -lt 15 |
Sort-Object FreePct |
Format-Table SystemName, DeviceID, FreeGB, SizeGB, FreePct -AutoSize
Run it daily and compare output. A volume that lost 5GB of free space in 24 hours deserves a page. A volume that has been at 88% for a year does not. In AlertMonitor, the per-device health baseline encodes exactly this difference, so the backup volume stops paging you and the shrinking OS volume does not get to wait.
3. Document what healthy actually means on your critical servers. Before you can judge whether an alert is meaningful, define the baseline — the services that must be running. This quick sweep catches the "it has been stopped since Tuesday" surprises:
$server = 'SRV-APP-03'
$critical = 'W3SVC','Spooler','MSSQLSERVER'
Get-Service -ComputerName $server -Name $critical |
Where-Object Status -ne 'Running' |
ForEach-Object { "ALERT: $server -> $($_.Name) is $($_.Status)" }
Load the resulting service list into AlertMonitor as service monitors. From then on, "service stopped" alerts fire only for services that actually matter, with the baseline attached to the alert.
4. Do the same across your Linux fleet without SSH-ing box by box:
for host in web01 db01 app01; do
printf '%s -> ' "$host"
ssh -o ConnectTimeout=5 "$host" "df -h / | tail -1"
done
Any host that times out or returns a root partition above your threshold belongs in monitoring — as a contextual alert, not a raw number.
5. Write the escalation policy down, then make the tool enforce it. On paper: P1 pages the primary, escalates at 10 minutes, hits the NOC manager at 25. Build exactly that as an escalation policy in AlertMonitor, scoped per client and severity. The point of policy automation is that it works at 2 AM, when humans do not.
6. Tie suppression windows to your patch schedule. Create the maintenance window (Saturday 22:00–04:00), scope it to the patch group, and link it to the patch job. Stop punishing yourself for doing maintenance correctly.
The Takeaway
When the fundamentals wobble, "add more AI" is a slogan, not a fix — ask Google's search team. Your on-call problem is the same shape. You do not need a machine-learning model to tell you a disk is filling up. You need the alert to tell you which disk, which client, how fast it is growing, whether that is normal for that server, whether it is mid-patch, who is next in the rotation if nobody acknowledges, and a ticket that opens itself with all of it attached.
That is signal quality. That is what every AlertMonitor alert is built on: fewer overnight pages, faster response, and an on-call rotation that stops burning out the people you can least afford to lose.
Related Resources
AlertMonitor Alert Management & On-Call Operations AlertMonitor Platform Overview Book a Demo Alert Management & On-Call Operations Resources
Is your security operations ready?
Get a free SOC assessment or see how AlertMonitor cuts through alert noise with automated triage.