The Register reported that British Transport Police spent £320,000 on a six-month live facial recognition pilot. The final tally: exactly one alert generated, and that single alert was a false positive. Zero correct matches. One innocent person flagged. £320,000 spent.
If you run infrastructure monitoring, an on-call rotation, or a NOC for an MSP, don't file this under privacy news. File it under alert quality failure — because it is the cleanest public illustration of a problem your team is probably living with right now: detection systems that either say nothing when something is wrong, or say the wrong thing loudly.
Silence and Noise: The Two Ways Alerting Dies
A detection system has exactly two failure modes, and the BTP pilot managed both at once:
- Silence: zero true positives across six months of continuous live scanning. The system ran and caught nothing real.
- Noise: the only signal it produced was false — and a single false positive from a system that produces nothing else erases whatever trust remained.
Most IT monitoring stacks exhibit the same two failure modes simultaneously, just distributed across tools. Your RMM quietly misses the degraded RAID volume on the file server, then compensates by paging the night tech four times because a backup job's schedule drifted by ten minutes. Both failures destroy the same asset: the on-call team's trust in the alerts. Once that trust is gone, technicians stop responding promptly, and the monitoring platform effectively stops existing — you are just paying license fees for dashboard wallpaper.
Ask yourself the BTP question about your own stack: over the last six months, how many incidents did your monitoring catch before an end user did? If the honest answer makes you uncomfortable, keep reading.
The Problem in Depth: Why Modern Monitoring Stacks Fail at Signal Quality
Static thresholds with no baselines
Most tools ship with defaults: alert at 90% CPU, 90% disk, low memory. On a SQL server that legitimately spikes to 95% during nightly ETL, that is a false-positive machine. On a file server whose data volume grew steadily from 80% to 88% over eight months, a static threshold means you get either months of silence followed by panic, or a year of "disk at 90%" alerts everyone learned to ignore. Neither is signal.
No deduplication
A flapping switch port, a disk dropping out of a RAID-5 array, a misconfigured backup job that fails every 30 minutes — typical RMM and monitoring platforms treat each recurrence as a brand-new alert. One root cause becomes 400 notifications in a night, and every one of them lands in the same queue as a potential emergency.
No maintenance window suppression
Patch Tuesday reboots take 30 servers down for eight minutes each. Without maintenance windows, that is 30 "device down" pages delivered to whoever is on call — during the exact window when they should be watching for a real problem, like a server that doesn't come back up.
Alerts with no context
"Disk critical on SRV-03." Which volume? Which client? What was it an hour ago — is this a growth trend or a logging loop gone wild? What does healthy look like for this device? In most stacks, the alert answers none of it, so the tech logs into the box at 2am to find out, one tool at a time.
Siloed architecture
This is the structural cause underneath all of the above. Monitoring bought from one vendor, RMM from another, helpdesk from a third, patching from a fourth — whether your stack is NinjaOne plus PRTG plus ConnectWise, or SolarWinds plus a standalone ITSM, silos are silos. Nothing links the alert to the device record, the device to the ticket, or the ticket to the fix. The MSP tech supporting one client has five tabs open across four tools just to answer "is this actually broken?" Every integration gap becomes minutes of context-switching per alert, multiplied by hundreds of alerts.
What it costs in real numbers
- Acknowledgment time: in a noisy queue, pages sit 30–60 minutes before a human decides they are real. That delay is added to every genuine incident.
- False positive economics: every false 2am page costs 20–30 minutes of night-shift tech time plus a permanent withdrawal from the trust account. After the tenth false page, the eleventh real one gets skimmed.
- SLA misses and credits: when a real alert drowns in known noise — buried under 60 weekly backup warnings from a client someone muted "temporarily" — the response clock doesn't start until a human notices. MSPs eat the credit; internal teams eat the escalation from the business.
- Burnout: technicians don't leave because of hard problems. They leave because of 3am pages that turn out to be nothing, again.
The scenario plays out weekly across the industry: a 40-client MSP's night shift mutes backup alerts for one client after months of noise. Three weeks later, that client's Exchange database fails at 11pm. The real alert fires, sits under a pile of known junk, and gets acknowledged 47 minutes later. The client's morning is wrecked, a service credit gets issued, and nobody can even measure the damage properly — because the alert data and the ticket data live in systems that don't talk.
How AlertMonitor Solves This: Signal Quality, Not Volume Reduction
AlertMonitor was built on a specific insight: alert fatigue isn't a volume problem — it's a signal quality problem. Muting things to cut noise just recreates the BTP failure mode (expensive silence). The goal is that every alert reaching a human is worth responding to, and every alert that isn't never does.
Here is what that looks like in the platform:
- Full context on every alert. Device, client, what changed, and what healthy looks like — attached to the alert itself. A disk alert in AlertMonitor tells you which volume, which client, the growth trend, and the baseline before anyone opens a terminal.
- Smart deduplication. A repeating failure or flapping interface collapses into one incident with a full timeline. The history exists for the post-mortem; it doesn't exist in the on-call queue as 400 pages.
- Maintenance window suppression. Planned reboots and change windows simply don't generate pages. Events are recorded; nobody's phone lights up at 2am for work you scheduled yourself.
- Configurable escalation with multi-level on-call routing. Alerts route to the right tech first — per client for MSPs, per system for internal teams — and escalate automatically if unacknowledged. No shared-inbox triage at 3am.
- One platform instead of four or five. Infrastructure monitoring, RMM, integrated helpdesk, patch management, and network topology mapping in a single product. Alert fires → device and its history are one click away → open a remote session or push a patch → ticket created and linked automatically.
The workflow, before and after
Fragmented stack: alert email arrives → log into the RMM and hunt for the device → remote in separately → open the helpdesk and create a ticket manually → paste notes between systems → update the client. Twenty to forty minutes per alert, most of it archaeology.
AlertMonitor: alert arrives with device, client, and context → one click to the device and its history → remediate via integrated RMM → auto-linked ticket documents everything. Minutes, not half an hour — and every step is captured for honest SLA reporting.
The measurable outcome: fewer overnight pages, faster acknowledge-to-resolve, and an alert-to-incident ratio you can actually report on. If your numbers ever start looking like BTP's — one alert, six months, and it was wrong — you will see it immediately instead of discovering it in a post-mortem.
Practical Steps: Fix Your Alert Quality This Week
1. Audit the last 30 days of alerts
Count total alerts and count the ones that required action. If the actionable rate is under 10%, you have a signal quality problem, not a staffing problem. This audit costs an afternoon; BTP's cost £320,000.
2. Baseline before you threshold
You cannot alert meaningfully on disk usage until you know what normal looks like per volume. Snapshot your Windows estate first:
# Establish a disk usage baseline across servers before setting thresholds
$servers = Get-Content .\servers.txt
Get-CimInstance -ComputerName $servers -ClassName Win32_LogicalDisk -Filter "DriveType=3" |
Select-Object @{n='Server';e={$_.PSComputerName}},
@{n='Drive';e={$_.DeviceID}},
@{n='FreeGB';e={[math]::Round($_.FreeSpace/1GB,1)}},
@{n='SizeGB';e={[math]::Round($_.Size/1GB,1)}},
@{n='FreePct';e={[math]::Round(($_.FreeSpace/$_.Size)*100,1)}} |
Sort-Object FreePct |
Export-Csv .\disk-baseline.csv -NoTypeInformation
Run it weekly for a month. Volumes that sit flat at 85% get a high threshold or get left alone; volumes climbing two points a week get a low one and a capacity ticket.
3. Hunt the repeat offenders
Your top 10 noisiest alerts usually trace back to three or four root causes. Services set to Automatic that legitimately stop during maintenance are a classic — alert only on the real failures:
# Flag auto-start services that are NOT running across a server list
$servers = Get-Content .\servers.txt
$critical = 'MSSQLSERVER','wuauserv','W3SVC','Spooler'
foreach ($srv in $servers) {
foreach ($svc in $critical) {
$s = Get-Service -Name $svc -ComputerName $srv -ErrorAction SilentlyContinue
if ($s -and $s.StartType -ne 'Disabled' -and $s.Status -ne 'Running') {
[PSCustomObject]@{
Server = $srv
Service = $svc
Status = $s.Status
Time = Get-Date -Format 'yyyy-MM-dd HH:mm'
}
}
}
}
4. Do the same on your Linux fleet
# Flag any mounted filesystem over 85% full — build thresholds from real data
df -h --output=source,pcent,target -x tmpfs -x devtmpfs | awk 'NR>1 && int($2) >= 85 {print $1, $2, $3}'
5. Put maintenance windows around every planned change
Patch reboots, vendor maintenance, migrations — if you scheduled it, it should never page anyone. In AlertMonitor, maintenance window suppression is configured per device or per client, and the suppressed events are still logged for review. You lose the noise without losing the record.
6. Wire alerts to tickets
Every page should produce a linked record you can report against. If your alerting and helpdesk live in separate systems with no link, that integration gap is the single highest-value thing to close — it is what makes SLA reporting possible and post-mortems honest.
The BTP pilot spent £320,000 to discover, after six months, that its detection system produced one wrong answer. You can run the equivalent audit on your monitoring stack for the cost of an afternoon — and unlike a national police force, you can fix it before the next Saturday-night outage finds out for you.
Related Resources
AlertMonitor Alert Management & On-Call Operations AlertMonitor Platform Overview Book a Demo Alert Management & On-Call Operations Resources
Is your security operations ready?
Get a free SOC assessment or see how AlertMonitor cuts through alert noise with automated triage.