TFG London — parent of Hobbs and Whistles — is the latest retailer to watch its IT systems go offline and its orders pile up unfulfilled, joining autumn's growing "pwned collection" of breached and broken retail estates. Whatever the root cause turns out to be, the operational story underneath the headline is one every IT pro knows by heart: a stack-wide outage where detection — not response — was the first thing to fail.
Because here's the uncomfortable question: if your estate went dark at 2 a.m. tonight, who finds out first — your on-call tech, or your customers?
The Problem: Your On-Call Is Drowning in Noise and Starved of Signal
Walk through what actually happens when a mid-sized estate has a serious event — ransomware, a failed SAN controller, a core switch stuck in a reboot loop. The trigger doesn't matter. The night does:
02:14 — The on-call phone buzzes. The shared alert mailbox shows 400+ emails since 01:50. The RMM flagged 60 stopped services across 12 servers. The standalone network monitor (PRTG, SolarWinds, Zabbix — pick your flavor) is screaming about unreachable SNMP targets. The helpdesk — ConnectWise, ServiceDesk Plus, whatever lives beside the RMM instead of inside it — has 38 new tickets, all some variation of "can't check out."
None of these systems talk to each other. So the first 45 minutes of the incident aren't spent fixing anything. They're spent doing manual correlation across five browser tabs to answer one question: how big is this, and where does it start?
This isn't a people problem. It's architecture:
- Siloed alert engines. RMM, network monitoring, and helpdesk each maintain their own alert logic, severity scales, and escalation rules — configured by different people at different times, reviewed never.
- No deduplication or correlation. One root cause — say, a failed uplink — fans out into hundreds of downstream child alerts. Every one of them pages somebody.
- Alert fatigue as a system feature. After weeks of noise, techs mute channels and skim emails. That works fine until the one real page is buried under 200 false ones — which is exactly how outages get discovered by customers and journalists instead of staff.
- Escalation rot. Policies were configured three years ago. Half route to a distribution list nobody reads. Some alerts suppressed for a maintenance window in March are still suppressed.
- No context at 3 a.m. "CPU 94% on SRV-APP-02" tells the on-call tech nothing. Which client? What's the baseline? What does healthy look like? Is this a page or a Monday morning?
The measurable cost: mean time to acknowledge stretches from minutes to hours. MTTR doubles because diagnosis starts from zero. SLA reporting is guesswork because ticket data and monitoring data live in separate systems with different clocks. And your best engineers start job hunting, because nobody signs up to be woken at 2 a.m. for the fourth false positive of the week. For a retailer like TFG London, the downtime itself is the visible symptom — stalled orders, abandoned carts, and a support backlog that takes weeks to clear.
How AlertMonitor Rebuilds the On-Call Signal Path
AlertMonitor was designed around a specific insight: alert fatigue isn't a volume problem — it's a signal quality problem. You don't fix it by monitoring fewer things. You fix it by making every alert worth acting on.
Full context on every alert. Each alert carries the device, the client, what changed, and what healthy looks like. "SQL-PROD01 disk F: crossed 90% — was 71% six hours ago, growing ~3GB/hr, projected full in 09:40" is a decision. "Disk warning" is an interruption.
Smart deduplication and correlation. When a root-cause event cascades, AlertMonitor collapses the child alerts into a single incident with an affected-asset list. Your on-call tech gets one page reading "CORE-SW-01 unreachable — 47 devices impacted" instead of 47 separate pages.
Multi-level on-call routing that stays maintained. Escalation policies are configured per client and severity, tied to rotation schedules, with automatic step-up if an alert isn't acknowledged. A P1 goes to the primary, then the secondary, then the escalation manager — automatically, in minutes. No distribution lists.
Maintenance windows that expire. Suppress alerts during planned change work, and the suppression auto-expires. No more forgotten mutes hiding a real failure three weeks later.
One platform, one workflow. Because monitoring, RMM, helpdesk, patch management, and network topology live in the same product, an alert flows straight through: the alert opens or links to a ticket, the ticket shows the device's health history and patch state, and the tech launches a remote session from the same pane. No tab juggling, no copy-pasting device names between systems.
The before and after, concretely: the old way is a tech paged 30 times in a night, spending 45 minutes establishing scope, fixing the fault, and reconstructing the timeline from memory afterward. The AlertMonitor way is one correlated page at 02:16 with scope already attached, remote action taken by 02:25, a ticket auto-populated with the alert timeline, and everyone else on the rotation left asleep. That's the difference between a 90-minute incident and a 9-minute one — and between a team that's burning out and one that isn't.
Practical Steps You Can Take This Week
1. Quantify your noise. Pull the last 30 days of alerts from every tool you run and count by source. In most estates, three or four checks generate 60–70% of all pages. Delete, downgrade, or fix them.
2. Apply the "so what" test. For every remaining alert, ask: if this fires at 3 a.m., what does the on-call tech do? If the answer is "nothing until morning," it shouldn't page anyone.
3. Verify your detection actually works. The checks behind your critical-service alerts should be tested, not assumed. Here's a quick sweep for core Windows services across your server estate:
# Verify critical services are running across core servers
$servers = 'ERP-APP01','SQL-PROD01','FS-01','DC-01'
$critical = 'MSSQLSERVER','DNS','Netlogon','W32Time'
Get-Service -ComputerName $servers -Name $critical -ErrorAction SilentlyContinue |
Where-Object Status -ne 'Running' |
Select-Object @{n='Server';e={$_.MachineName}}, Name, Status |
Format-Table -AutoSize
Wire this into AlertMonitor as a scheduled remote check on each server. When it fails, the alert arrives with the server, the service, and its state — not a generic "script check failed."
4. Catch the quiet killers before they page you. Disk exhaustion is still the most avoidable outage in IT:
# Flag any fixed volume under 15% free across the estate
$servers = 'FS-01','SQL-PROD01','RDS-04','APP-02'
Get-CimInstance -ClassName Win32_LogicalDisk -Filter 'DriveType=3' -ComputerName $servers |
Select-Object @{n='Server';e={$_.SystemName}}, DeviceID,
@{n='FreeGB';e={[math]::Round($_.FreeSpace/1GB,1)}},
@{n='FreePct';e={[math]::Round(($_.FreeSpace/$_.Size)*100,1)}} |
Where-Object FreePct -lt 15 |
Sort-Object FreePct
In AlertMonitor, thresholds are per-device with trend data behind them, so FS-01's data volume paging at 85% doesn't drag the scratch volume into the conversation.
5. Give your Linux boxes the same treatment:
# Service health sweep — the logic behind a "critical service down" alert
for svc in nginx postgresql redis-server; do
if systemctl is-active --quiet "$svc"; then
echo "OK: $svc up since $(systemctl show -p ActiveEnterTimestamp --value $svc)"
else
echo "DOWN: $svc — last log lines:"
journalctl -u "$svc" -n 5 --no-pager
fi
done
6. Test the pager path monthly. Fire a synthetic P1 at 3 p.m. on a Friday. Who got buzzed? How fast did they acknowledge? If nobody noticed until Monday, your escalation ladder is decoration.
The Takeaway
The TFG London story will be told as a security story, and reasonably so. But for the people running the estate, the lesson lands on the operations side: the quality of your incident response was decided months before the incident — in how your alert pipeline is configured. Whether the trigger is ransomware or a dead switch, the on-call team's night comes down to one question: did the right page reach the right person, with enough context to act?
If the honest answer today is "our on-call usually finds out from users," that's fixable — this quarter, not next year. Consolidate the signal, route it intelligently, and let the platform carry the context so your people can carry the response.
Related Resources
AlertMonitor Alert Management & On-Call Operations AlertMonitor Platform Overview Book a Demo Alert Management & On-Call Operations Resources
Is your security operations ready?
Get a free SOC assessment or see how AlertMonitor cuts through alert noise with automated triage.