One Cut Cable, 400 Alerts at 2 A.M.: Fixing the Upstream Outage Alert Storm

Whitehall has put a 14-year prison sentence on the table for sabotaging UK subsea cables, alongside new powers over telecoms operators, as officials react to a string of suspicious cable incidents in the Baltic and beyond. The subtext is impossible to miss: the physical pipes that carry the vast majority of intercontinental data traffic — commonly cited at 95% or more — are now treated as critical national infrastructure, and the operators who run them are being held to a demonstrable standard of detection and response.

If you run IT for a company or an MSP, you will never be the one prosecuting a saboteur. But you will absolutely be the one holding the phone when a cable gets cut — and it doesn't take an act of hostile intelligence. An anchor drag, a trawler, a burst duct, a JCB through a fiber cabinet in an industrial estate: upstream transport fails somewhere, in someone's dependency chain, every single day.

And here's the part that should sting: when it happens, your monitoring stack is more likely to bury your on-call technician under a blizzard of noise than to tell them what actually broke. Not because the tools are broken — because they were never designed to understand that 400 alerts can share one cause.

What Actually Happens When the Pipe Breaks

It's 01:42 on a Sunday. A vessel drags its anchor across a cable, and your firewall's primary WAN circuit dies. Failover kicks in — LTE or a secondary circuit — but with higher latency and real packet loss.

Over the next eight minutes, on a typical fragmented stack, here's what fires:

• Standalone monitoring (PRTG, SolarWinds, Zabbix, Nagios): the ICMP sensor on the WAN interface goes red, and every "host unreachable" sensor behind the firewall follows it — say 25 devices.
• RMM platform (NinjaOne, ConnectWise RMM, Datto RMM): each offline agent generates its own offline alert. At a 40-workstation site, that's 40 more.
• Backup software: the nightly Veeam job can't reach the offsite repository and fails.
• Windows Server checks: DNS timeout warnings, AD replication warnings, DFSR backlog alerts.
• Helpdesk: 30 users report Teams, the ERP, or the phone system is down. Tickets start landing at 01:47.
• VoIP monitoring: extension registration failures cascade in.

Total: somewhere between 150 and 400 notifications, tickets, and emails from exactly one physical event. Median time to acknowledge balloons — not because the on-call tech is slow, but because the one alert that matters is buried under page #180, and every channel (email, SMS, push, ticket queue) is lit up at once.

Worse is what happens the second and third time. After a couple of these storms, technicians learn to bulk-acknowledge, snooze, or route alert emails into a folder nobody reads. That's when the genuinely dangerous scenario appears: an independent, unrelated failure — a degraded RAID array, a failed PSU — lands in the middle of a WAN storm and gets acknowledged along with the rest of it. I've watched a storage incident sit for 40 minutes inside a network alert storm because it "looked like part of the outage." Downtime doubled, and the post-mortem was ugly.

Why the Tools Behave This Way

This isn't a training problem. It's architecture.

Every device polls independently. Classic monitoring platforms treat each sensor as an isolated check. There is no shared model that says "these 25 hosts sit behind this firewall, on this circuit." Parent-child dependencies, where they exist at all, are manual per-device config that nobody maintains across hundreds of endpoints.

RMM and monitoring don't talk. The RMM knows the agent is offline; the monitoring platform knows the firewall WAN is down; neither tells the other. Correlation happens in a human's head, at 2 a.m., across five browser tabs.

The helpdesk is a separate system. User-reported symptoms land in a PSA that has no idea a monitoring event exists, so the incident spawns 30 tickets, and your SLA clock starts when the first user called — long before anyone on the technical side acknowledged anything.

Alert payloads carry no context. "Device offline — OFFICE-WS-23" is not a signal; it's a fragment. No client, no site, no parent event, no picture of what healthy looked like an hour ago.

The operational cost adds up like this: MTTR stretches because "time to understand" dwarfs "time to fix." SLA reports are fiction, because incident data and ticket data live in different systems. And on-call engineers burn out — the turnover cost of a night-shift tech who quit because "the phone never stopped" is rarely attributed to the monitoring tooling, but it should be.

The regulatory signal in the UK story is a leading indicator for all of us. Governments are moving from "connectivity is a commodity" to "connectivity resilience must be demonstrable." When an auditor or a client asks you to show how a transport failure was detected, acknowledged, and escalated, "our monitoring sent 400 pages" is not an answer.

How AlertMonitor Changes the 2 A.M. Equation

AlertMonitor was built on a specific insight: alert fatigue is not a volume problem — it's a signal quality problem. Applied to the cable-cut scenario, it changes the timeline like this.

One root cause, one event. Smart deduplication groups the cascade. The firewall WAN failure becomes the parent event; the 25 "host unreachable" alerts and the 40 offline agents attach to it instead of paging independently. The on-call tech gets one page, not 180.

Alerts that read like a handover note. Every alert carries full context — device, client, what changed, and what healthy looks like. The 01:42 page says: Client A, Site 1 — primary WAN circuit down since 01:42; failover active on WAN2; 23 downstream devices unreachable as a consequence; baseline before the event: 4 ms latency, 0% loss. The tech doesn't reconstruct the incident — they verify failover and open the carrier ticket.

Topology-aware blast radius. Because network topology mapping is built into the platform, AlertMonitor already knows which switches and hosts sit behind which firewall and circuit. Downstream unreachability is inferred from the map, not discovered one red sensor at a time.

Routing that respects roles and clients. Multi-level on-call routing sends the WAN event to network on-call — not the application team, not the helpdesk lead. For MSPs this matters even more: when one metro fiber cut touches six clients, you get six clean, per-client incidents in the NOC queue — not 600 undifferentiated pages.

Maintenance windows that actually suppress. The carrier announces a 01:00–05:00 circuit upgrade? Schedule the window once, and planned work never pages anyone. Suppression is logged, not invisible, so nothing gets silently lost.

The helpdesk is the same system. The incident auto-creates one ticket with the full alert timeline attached. The 30 user tickets that trickle in get merged into it. Your SLA report finally measures the incident from detection to resolution — because monitoring and ticketing share one data model.

The old way: five tabs, 25 minutes of manual correlation, first meaningful action well past 02:00. With AlertMonitor: one page at 01:42 with root cause and blast radius, failover confirmed and carrier ticket opened in about 90 seconds. Everyone else sleeps.

Practical Steps You Can Take This Week

1. Map your dependencies before the next cut. You can't deduplicate what you haven't modeled. In AlertMonitor, build the topology so parent-child relationships between circuits, firewalls, and downstream hosts are explicit. If you're not there yet, a spreadsheet mapping each site to its circuits and edge devices is still better than nothing.

2. Monitor the edge, not just the interior. Most teams alert heavily on internal device state and almost not at all on the transport itself. Add checks on WAN interface state and round-trip latency to both circuit gateways. A basic failover watchdog from a server at the site:

# wan-failover-check.ps1 — scheduled task, every 5 minutes
$primaryGw   = "203.0.113.1"    # primary circuit handoff
$secondaryGw = "198.51.100.1"   # backup circuit / LTE router
$log         = "C:\Logs\wan-check.csv"

function Test-Circuit {
    param([string]$Target)
    $pings = Test-Connection -ComputerName $Target -Count 4 -ErrorAction SilentlyContinue
    [pscustomobject]@{
        Time      = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
        Target    = $Target
        Reachable = [bool]$pings
        AvgMs     = if ($pings) {
                        [math]::Round(($pings | Measure-Object -Property ResponseTime -Average).Average, 1)
                    } else { $null }
    }
}

$results = (Test-Circuit $primaryGw), (Test-Circuit $secondaryGw)
$results | Export-Csv -Path $log -Append -NoTypeInformation

if (-not $results[0].Reachable -and $results[1].Reachable) {
    Write-Output "FAILOVER: primary circuit down, backup healthy"
    exit 2
}
elseif (-not $results[0].Reachable -and -not $results[1].Reachable) {
    Write-Output "CRITICAL: total WAN loss on both circuits"
    exit 3
}
else {
    Write-Output "OK: primary $($results[0].AvgMs) ms, backup $($results[1].AvgMs) ms"
    exit 0
}

The exit codes matter: wire them into your scheduler or RMM so "failover active" is a warning-level event and "total loss" is critical. That gives you one meaningful edge alert instead of forty agent-offline pages that follow it.

3. Watch the interface itself on Linux-based gear. Error counters climb before a link dies outright:

#!/usr/bin/env bash
# wan-health.sh — interface state and error counters
IFACE="eth0"
STATE=$(cat "/sys/class/net/${IFACE}/operstate")
RX_ERR=$(cat "/sys/class/net/${IFACE}/statistics/rx_errors")
TX_ERR=$(cat "/sys/class/net/${IFACE}/statistics/tx_errors")

echo "${IFACE}: state=${STATE} rx_errors=${RX_ERR} tx_errors=${TX_ERR}"

if [ "${STATE}" != "up" ]; then
    logger -t wan-health "CRITICAL: ${IFACE} is ${STATE}"
    exit 2
fi
exit 0

Run it from cron every five minutes and diff the counters — a rising rx_errors trend on an interface that is technically "up" is the early warning that the circuit or the SFP is on its way out.

4. Make your escalation policy reflect dependency, not just severity. The policy shape you want: a root cause that swallows its own cascade and escalates on a clock.

# Dependency-aware escalation policy (illustrative shape)
policy: wan-upstream-failure
trigger:
  check: wan-interface-status
  device_role: [firewall, edge-router]
deduplication:
  window: 15m
  group_by: [client, site, circuit]
suppress_children:
  when_parent_failed: host-unreachable
escalation:

• after: 0m route: network-oncall
• after: 15m route: network-team-lead
• after: 30m route: incident-manager maintenance_window: suppress: true

In AlertMonitor this is configuration, not code — the deduplication window, grouping keys, suppression rules, and escalation levels are all set in the policy editor, per client and per site.

**5. Register carrier maintenance windows.** Every circuit has an upstream provider announcing planned work. Put those windows into maintenance suppression once, and the "planned downtime" page — which is pure noise — disappears from your rotation.

**6. Measure signal quality, not alert volume.** After the next transport incident, count the alerts: how many paged, how many were deduplicated into the root cause, and MTTA before and after. AlertMonitor's incident timeline shows exactly that split, so "we cut overnight pages by 80%" becomes a report you can hand your IT manager or your client — and, increasingly, an auditor.

## The Bottom Line

Governments are ratcheting up the stakes on the physical layer, and subsea cable sabotage is now a 14-year felony in the UK. Whether the next cut near your sites is sabotage, an anchor drag, or a digger in the wrong place, the difference between a calm three-minute failover response and a burned-out on-call rotation isn't the cable. It's whether your alerting can collapse 400 symptoms into one signal — with the device, the client, and what healthy looks like attached.

## Related Resources

[AlertMonitor Alert Management & On-Call Operations](https://alertmonitor.ai/platform/alert-management)
[AlertMonitor Platform Overview](https://alertmonitor.ai/platform)
[Book a Demo](https://alertmonitor.ai/contact)
[Alert Management & On-Call Operations Resources](https://alertmonitor.ai/blog/alert-management)