An AT&T retail store employee was recently sentenced to 16 months in prison for a SIM-swap side hustle, as reported by The Register. His cut of the scheme: less than $4,000. The intended losses across the attacks: roughly $600,000. No exploit kit, no zero-day — just a trusted person at a trusted desk flipping a trusted channel.

If you run IT operations — internal team or MSP — read that twice, because it describes the architecture of most alerting setups with uncomfortable accuracy: monitoring platform → SMS or email-to-SMS gateway → one on-call phone number → one tired human. Every hop is trusted. Almost none of it is verified. When a hop fails — carrier throttling, a number porting error, a muted phone, a dead battery — the failure is silent. Your monitoring console logs the alert as sent. The human never sees it. And in far too many tools, the escalation rule then fires and re-sends the alert down the exact same dead path.

The case carries a second lesson: valuable targets fall because the people protecting them have been drowned in noise until they stop verifying anything. An on-call engineer processing 400 alerts a night doesn't investigate delivery channels — they mute the phone. Then the one alert that mattered gets muted with the other 399.

This post is about closing both gaps: making alert delivery verifiable instead of assumed, and making the signal rare and rich enough that nobody ever wants to mute it.

The Problem in Depth: "Sent" Is Not "Delivered" Is Not "Seen" Is Not "Acknowledged"

Single-channel delivery is a silent single point of failure

Most stacks — PRTG feeding an email-to-SMS gateway, NinjaOne or Datto RMM notifying a paging app tied to one cell number, Zabbix emailing a distribution list — treat an alert as done when it leaves the system. None of them answer the only question that matters at 2 AM: did a human acknowledge this?

The failure modes are unglamorous and real:

• The carrier throttles bulk SMS from your gateway and holds your page in a queue for 47 minutes.
• The on-call tech's phone is on Do Not Disturb during a family dinner; the monitoring tool has no idea.
• A number port during a carrier migration silently re-routes your pages for a week.
• The email-to-SMS gateway's SMTP relay certificate expires on a Saturday.

In every case, the monitoring system shows a successful send. There is no error to alert on, because the alerting layer itself is what broke. It's the operational equivalent of the insider problem: the failure happens at the trusted layer, where nobody is looking.

Alert fatigue trains humans to disable the safety net

Watch any NOC queue for a week and count. A disk sitting at 88% fires an alert at 88%, then 89%, then 90%, every night. A flapping VPN tunnel pages 30 times before sunrise. The average tech handles hundreds of notifications a month where the correct response was "nothing." So people do the rational thing: inbox rules, muted apps, phone face-down. MTTA drifts from 4 minutes to 40. When the real alert finally fires — a domain controller's system disk hits 100% at 1:38 AM — it arrives dressed exactly like the 397 false alarms before it.

Fragmented tools hide the blast radius

The alert says "SRV-DB-03: free disk space below 5%." Which client? Domain controller or utility box? What SLA? Who owns it? In the typical three-tool stack — monitoring here, ConnectWise or Autotask helpdesk there, RMM somewhere else — answering those questions means five tabs, three searches, and guessing client priority from memory. Triage that should take 20 seconds takes 10 minutes, and every escalation decision is made with half the context. Meanwhile, the IT manager builds MTTR and SLA reports by hand-stitching monitoring timestamps to helpdesk tickets — two systems that don't even agree on what "acknowledged" means.

What it costs, in numbers you recognize

A composite of real MSP incidents: a 60-endpoint client, one on-call tech, SMS-only paging. The DC's system disk starts filling at 1:38 AM after a log spiral. The page is throttled and lands at 2:31 AM on a phone in DND. The escalation policy re-sends SMS every 30 minutes — same carrier, same path. At 8:02 AM the client's office manager calls because nobody can log in. Time to acknowledge: 6 hours, 24 minutes. Outcome: two SLA credits, a finance team locked out at quarter-end, and an on-call tech spending his morning explaining why he slept next to a phone that never rang.

Internal IT lives the same story: an admin's cell was ported during a carrier migration and a week of after-hours pages vanished. Nobody found out from monitoring. They found out from user complaints.

How AlertMonitor Solves This

AlertMonitor was designed around a specific insight: alert fatigue isn't a volume problem — it's a signal quality problem, and signal quality includes whether the signal reliably reaches a human who can act.

• Delivery is confirmed, not assumed. Every alert is tracked through sent → delivered → acknowledged. No acknowledgment within the configured window triggers the next escalation step.
• Escalation paths are independent and multi-level. A P1 routes: push to primary on-call → SMS plus automated voice call after 5 minutes without ack → secondary on-call → team lead. Those channels don't share infrastructure, so one dead path can't black-hole a page the way "re-send the SMS" does.
• Every alert carries full context. Device, client, what changed, and what healthy looks like — in the notification itself. From the alert you open the linked helpdesk ticket, launch an RMM remote session, or check patch status. Triage in seconds, not across five tabs.
• Smart deduplication kills the noise that trains humans to mute you. "Disk 91%, 92%, 93%" becomes one alert that escalates as the condition worsens instead of a nightly series.
• Maintenance window suppression means planned patching pages nobody — suppression is explicit, scheduled, and logged, not a silenced phone.
• Escalation policies are configurable per client, severity, and time of day. A P1 for a financial client at month-end routes differently than a P3 printer alert on a Saturday, because those are different business situations.
• The audit trail is the SLA report. Who was notified, on which channel, when they acknowledged — captured in the same system that generated the alert. No manual stitching, no fictional reports.

Old way: monitoring emails an SMS gateway, hope does the rest, and the helpdesk learns about the outage from the client. AlertMonitor way: contextual alert, confirmed delivery, acknowledgment in minutes, ticket auto-linked, remote session one click away.

Practical Steps You Can Take Today

1. Map every critical alert to its physical delivery path

Take your top 20 alert rules and trace, hop by hop, how each notification reaches a human. Count the independent pieces of infrastructure between "problem detected" and "eyeballs." If the answer for any rule is "one," you've found your SIM-swap moment.

2. Verify the delivery infrastructure answers before you trust it

Your alert email path depends on an SMTP relay; your webhook path depends on an HTTP endpoint. Check both like you'd check any production service:

# Verify the SMTP relays your monitoring platform uses for alert email are reachable
$relays = "smtp1.corp.local", "smtp2.corp.local"
$results = foreach ($relay in $relays) {
    $tcp = Test-NetConnection -ComputerName $relay -Port 25 -WarningAction SilentlyContinue
    [PSCustomObject]@{
        Relay      = $relay
        Port25Up   = $tcp.TcpTestSucceeded
        ResolvedIP = $tcp.RemoteAddress
    }
}
$results | Format-Table -AutoSize
# Any row with Port25Up = False is a dead leg in your alert delivery chain

bash

From the monitoring server: confirm your paging webhook endpoint answers

A healthy endpoint returns 200 in well under a second

curl -s -o /dev/null -w "HTTP %{http_code} in %{time_total}s" https://alerts.yourcompany.com/health

3. Sanity-check the thresholds generating your noise

Before blaming your techs for muting notifications, look at what the thresholds actually fire on. Disk alerts are the usual offender:

# Pull disk free space across servers to sanity-check your disk alert thresholds
$servers = Get-Content .\servers.txt
$report = foreach ($server in $servers) {
    Get-CimInstance -ComputerName $server -ClassName Win32_LogicalDisk -Filter "DriveType=3" |
        Select-Object @{n='Server';e={$server}},
                      @{n='Drive';e={$_.DeviceID}},
                      @{n='SizeGB';e={[math]::Round($_.Size/1GB,1)}},
                      @{n='FreeGB';e={[math]::Round($_.FreeSpace/1GB,1)}},
                      @{n='FreePct';e={[math]::Round(($_.FreeSpace/$_.Size)*100,1)}}
}
$report | Where-Object { $_.FreePct -lt 15 } | Sort-Object FreePct
# If half the fleet sits at 12-14% free, a 15% threshold pages every night and means nothing

4. Confirm your agents are alive — no agent, no alert

An alert about a dead agent arrives exactly never. Spot-check a sample of endpoints:

# Confirm the monitoring agent is running and has heartbeated recently
systemctl is-active alertmonitor-agent
journalctl -u alertmonitor-agent --since "2 hours ago" | tail -5

5. Configure acknowledgment-required escalation in AlertMonitor

For each severity tier, set an acknowledgment window and an independent channel ladder:

• P1: push immediately → SMS plus voice call at 5 minutes without ack → secondary on-call at 10 → team lead at 15
• P2: push → email at 15 minutes without ack
• Deduplicate repeating conditions into one escalating alert rather than a nightly series
• Put planned maintenance into maintenance windows so suppression is explicit and auditable

6. Run a quarterly delivery drill

Once a quarter, fire a test page at each severity during real on-call hours and measure time-to-ack for every channel in the ladder. A channel that takes more than five minutes or fails without an error is your next 2 AM outage — discovered on a night when it costs you nothing.

The Takeaway

The AT&T clerk needed no zero-day — just trusted, unverified access to one channel, and people too buried in noise to double-check anything. Your alerting deserves better assumptions: delivery that is confirmed, escalation that doesn't share a failure domain with the thing that broke, and alerts rich and rare enough that nobody ever wants to mute them. That's the difference between a tool that says "sent" and an on-call operation that knows a human is on it.

Related Resources

AlertMonitor Alert Management & On-Call Operations AlertMonitor Platform Overview Book a Demo Alert Management & On-Call Operations Resources