When One Failure Becomes 400 Alerts: On-Call Lessons From the UNM Health Sciences Center Cyberattack

Day 2 of the University of New Mexico cyberattack: classes canceled again, the Health Sciences Center working through recovery in phases, and hospital operations that had initially gone to ambulance diversion cautiously accepting patients again. From the outside, it's a cybersecurity story. If you've ever been on call during a cascading infrastructure failure, you know it's also an alert-management story — and probably a story about the worst night of someone's on-call rotation.

Because somewhere in that ops room, in the first minutes, there was a console full of alerts. Hundreds of them. Authentication failures, unreachable hosts, failed backups, service crashes, VPN drops — all firing at once, all screaming, none of them labeled "this is the actual thing."

That's the moment that separates IT teams that contain an incident in minutes from teams that spend two days in phased recovery. And it's the moment most monitoring stacks are genuinely bad at.

The Reality of a Major Incident: The Alert Storm

Here's what actually happens on a monitoring console when something serious hits a network — a cyberattack, but equally a failed core switch, a dead SAN controller, or a botched hypervisor update:

• Every server that can't reach a domain controller throws authentication alerts.
• Every application that depends on those servers throws "unavailable" alerts.
• Every backup job against those servers fails and alerts.
• Every network path traversing the degraded segment reports packet loss or timeout.
• Every agent that can't check in generates its own "agent offline" alert.

One root cause. Four to five hundred alerts. Tools like Nagios, PRTG, Zabbix, and SolarWinds poll checks independently, so each one fires its own notification. Nothing tells the on-call tech that all 400 of these are one event.

So what does the on-call engineer do at 2 a.m.? They start triaging alert #1, then #2, then #7 — usually in the order the emails arrived, which has zero relationship to root cause. Meanwhile the real problem keeps spreading, and the tickets start arriving in the helpdesk: "I can't log in." "The system is slow." "Is the network down?"

Why Existing Tools Fail Here

Monitoring, RMM, and helpdesk live in separate worlds. Your monitoring tool (say, PRTG) emails an alert. Your RMM (ConnectWise Automate, N-able, NinjaOne) has its own alert list. Your helpdesk (ConnectWise Manage, ServiceNow, Freshservice) is collecting user tickets. During an incident, someone has to manually correlate all three — usually in different browser tabs, sometimes from home over VPN. The ticket that says "can't open patient records" never gets linked to the monitoring alert that would have explained it.

Alerts carry no context. "Host HSC-APP-14 is down" tells you nothing. Down how? Since when? What changed? Was it healthy an hour ago? Is this the first host or the fifteenth? Without context, every alert is a fresh investigation, and at 2 a.m. every investigation costs 15 minutes you don't have.

Escalation is a static email chain. In a lot of shops, "escalation policy" means the alert goes to a distribution list and you hope someone's phone is on. No acknowledgment tracking. No automatic escalation when nobody responds. For MSPs, often no per-client routing at all — your night tech gets paged for a client whose contract says next-business-day.

Planned noise isn't suppressed. Patch Tuesday rolls out, 60 servers reboot, and every one of them generates a "server down" page. Your on-call tech learns, permanently, that pages can be ignored — which is exactly the reflex that fails them when the page is real. Industry surveys have repeatedly found IT teams disregard a third or more of their alerts as noise. That number isn't a discipline problem. It's a tooling problem.

What It Actually Costs

• Detection and response time. When the signal is buried, minutes-to-detect becomes hours-to-days. In healthcare, that's not just an SLA miss — that's ambulance diversion.
• Ticket volume and duplication. Users report what monitoring already saw, in a different system, with no link between them. Your helpdesk drowns while your NOC drowns, separately.
• Burnout and turnover. Nobody quits over a well-handled incident. People quit over the fourth consecutive night of false alarms, followed by the one real alert nobody trusted.
• Reporting that can't be trusted. The IT manager writing the post-incident report can't build it, because the monitoring timeline lives in one tool, response actions in another, and tickets in a third. MTTR becomes a guess.

How AlertMonitor Handles the Storm

AlertMonitor was built around a specific insight: alert fatigue isn't a volume problem — it's a signal quality problem. You don't fix 2 a.m. by sending fewer emails; you fix it by making sure the one alert that matters carries everything the responder needs.

Smart deduplication and correlation. When one failure cascades, AlertMonitor collapses the storm into a single incident. The 400 "unreachable" alerts become one event with a timeline showing what fired first — which is almost always your root cause. The on-call tech answers one page, not four hundred.

Context on every alert. Each alert carries the device, the client, what changed, and what healthy looks like. Not "host down" but "HSC-APP-14: service stopped, host otherwise healthy for 214 days, baseline attached, change logged 6 minutes ago." The responder starts with a diagnosis, not a question.

Multi-level escalation that actually escalates. Tier 1 gets 15 minutes to acknowledge. No ack? It routes to tier 2. Then to the service owner. Per-client policies for MSPs mean a 24/7 client pages the night rotation while a business-hours client waits for morning — automatically, with no humans forwarding emails.

Maintenance window suppression. Planned reboots, patch nights, and phased recovery work don't generate pages. And here's the part most tools get wrong: AlertMonitor suppresses the expected noise while still alerting on the unexpected — a server inside the patch window that doesn't come back up still pages someone.

One platform: alert → action → ticket. Because monitoring, RMM, and helpdesk live in the same product, the workflow becomes: alert fires with full context → one click opens a remote session → run the remediation script → ticket auto-created and linked to the alert timeline. No tab switching, no copy-pasting device names between systems, no "which tool did we see this in first?" And when the incident's over, the post-incident report — detection time, response time, every action taken — already exists, because it all happened in one system.

For an MSP running a NOC, that's the difference between a tech juggling 12 tabs across 5 tools per client and a tech running one console across every client.

What You Can Do Today

Even before you change tools, you can reduce the next storm's damage. Start here.

1. Measure your noise. Pull the last 30 days of alerts and count. If your top ten checks generate most of your volume, you've found your deduplication and threshold problem. (If you can't produce that report in ten minutes, that itself is a finding.)

2. Kill the classic 2 a.m. disk page by catching disks early. A "disk full" page at 2 a.m. should never happen, because disk growth is slow and predictable. Run this across your Windows estate weekly:

$servers = Get-Content .\servers.txt
Get-CimInstance -ClassName Win32_LogicalDisk -Filter 'DriveType=3' -ComputerName $servers |
    Select-Object PSComputerName, DeviceID,
        @{n='SizeGB';e={[math]::Round($_.Size/1GB,1)}},
        @{n='FreeGB';e={[math]::Round($_.FreeSpace/1GB,1)}},
        @{n='FreePct';e={[math]::Round($_.FreeSpace/$_.Size*100,1)}} |
    Where-Object { $_.FreePct -lt 15 } |
    Sort-Object FreePct |
    Export-Csv .\low_disk_report.csv -NoTypeInformation

Fifteen percent free is your early warning. Set the ticket threshold at fifteen and the page threshold at five, and the 2 a.m. disk page disappears.

3. Verify critical services are actually running — especially after change windows. This catches the rebooted-but-not-recovered server before users do:

$critical = 'Netlogon','DNS','W32Time','wuauserv'
foreach ($srv in (Get-Content .\servers.txt)) {
    Get-Service -Name $critical -ComputerName $srv -ErrorAction SilentlyContinue |
        Where-Object { $_.StartType -eq 'Automatic' -and $_.Status -ne 'Running' } |
        Select-Object @{n='Server';e={$srv}}, Name, Status
}

In AlertMonitor, these become monitored checks with baselines and escalation policies attached — the script is your proof of concept; the platform is where it lives permanently.

4. Do the same for your Linux fleet:

#!/bin/bash
# Flag any host whose root filesystem is above 85 percent
for host in $(cat linux_servers.txt); do
    pct=$(ssh -o ConnectTimeout=5 "$host" "df --output=pcent / | tail -1 | tr -dc '0-9'")
    if [ "${pct:-0}" -gt 85 ]; then
        echo "WARNING: $host root filesystem at ${pct}%"
    fi
done

5. Know your patch state before you need it. During phased recovery — or any major incident — "what's patched and what isn't" is one of the first questions leadership asks. Have the answer ready:

Get-HotFix |
    Sort-Object InstalledOn -Descending |
    Select-Object -First 15 HotFixID, Description, InstalledOn

6. Define escalation tiers that match reality, not org charts. A sane default: tier 1 (on-call tech) gets 15 minutes; tier 2 (senior admin) gets 15 more; tier 3 (service owner or vendor) gets paged at the 30-minute mark for anything customer-impacting. Write it down, then enforce it with a tool — not with hope.

7. Put every planned change in a maintenance window. Patch night, migrations, phased recovery work — all of it goes on the calendar with suppression enabled. Your page rate drops immediately and, more importantly, every remaining page becomes believable again.

8. Write runbooks for your top five pagers. Whatever alerts most often — disk, a specific service, backup failures, certificate expiry — deserves a documented first response. The 3 a.m. responder shouldn't be inventing the remediation from scratch.

The Takeaway

UNM's recovery proceeding "in phases" is the right operational discipline — you can't restore everything at once, and you shouldn't. But phased recovery only works if your monitoring can tell you, per phase and per system, when something is genuinely healthy again rather than merely "not red anymore." That requires baselines, context, and one system of record for the whole incident.

Most IT teams will never face a hospital-grade cyberattack. Every IT team will face the alert storm that comes with a major failure — and the teams that come through fastest are the ones where the right person saw the right signal in the first minutes, with enough context to act. Fewer overnight pages. Faster response. A team that isn't burned out by its own monitoring tools. That's not a feature list; that's the whole point.

Related Resources

AlertMonitor Alert Management & On-Call Operations AlertMonitor Platform Overview Book a Demo Alert Management & On-Call Operations Resources