7.49 Million Files, Zero Pages: What the CenterPoint Energy Breach Teaches IT Teams About Alert Fatigue

CenterPoint Energy has confirmed that an intruder got away with customer information — and the Texas utility learned about it the way too many organizations learn about their worst incidents: from a forum post claiming 7.49 million files were up for grabs. Not from an alert. Not from an on-call page. Not from any system the company controls.

If you run infrastructure for a living, you have lived a smaller version of this story. The degraded RAID array that sat unattended all weekend because its alert went to a distribution list nobody reads. The SQL box whose transaction logs quietly filled the disk while "warning" emails piled up in a spam folder. The check someone silenced during a maintenance window three weeks ago and never re-enabled. The breach is the nightmare version. Alert blindness is the everyday disease — and most IT teams have it in some form.

The Problem in Depth: The Signal Existed, the Action Didn't

Here's the uncomfortable truth about incidents like this: in most environments, something was probably watching. The question is whether what it saw ever became a page a human answered. Most IT stacks — a standalone monitor here, an RMM there, a helpdesk over in another tab — fail at that handoff in three structural ways.

1. Alerts without context. Legacy monitoring and per-device RMM alerts fire on thresholds: "Disk C: 91% on SRV-APP12." Fine. But that alert doesn't tell the on-call tech what normal looks like, whether the disk is trending toward full in four hours or four weeks, whether the failed backup job at midnight is related, or what else on that host is degraded. So the tech opens four browser tabs, RDPs in, and triages by hand. At 2am. For the fifth time this month. The predictable result: someone downgrades that check to email-only. The black hole grows.

2. No deduplication, so alert storms burn out the humans. One flapping core switch at a client site takes connectivity down for 40 endpoints. The RMM does exactly what it's configured to do: 40 endpoint-down alerts, 20 firewall events, 12 agent-offline notifications, cascading over 90 minutes onto the on-call phone. After the third night of that, a tech installs a quiet-hours mute rule at midnight. Which is precisely when the domain controller develops its real problem — unnoticed. The tools didn't fail to detect anything. They detected forty times over. There was noise, but no signal.

3. Escalation that doesn't escalate. On-call rotations live in a spreadsheet or an Outlook calendar. A critical alert fires, the primary is asleep with the phone on Do Not Disturb — and that's the end of the chain. No acknowledgement requirement, no second tier, no fallback. Security researchers have documented median intrusion dwell times measured in weeks, historically months, before victims notice on their own. CenterPoint reportedly found out because criminals advertised the loot. That's not a monitoring gap; that's a notification-path failure.

The operational impact is measurable, and every sysadmin recognizes it:

• MTTR balloons because detection time dominates resolution. If a service dies at 11pm and nobody notices until 8am, the actual fix — often five minutes of work — is buried under nine hours of not knowing.
• End users become your monitoring system. Every user who notices the file server is slow becomes a human sensor filing a ticket. Your helpdesk queue fills with symptoms instead of causes, and every duplicate ticket burns 10–15 minutes of triage.
• SLA reporting is fiction. When monitoring data lives in PRTG, tickets live in ConnectWise Manage, and the rotation lives in a spreadsheet, nobody can answer "how long did critical incidents actually take last quarter?" without a week of Excel archaeology.
• Burnout drives attrition. On-call techs trained by noise learn to ignore pages — including the real ones. MSPs see it as technician churn. Internal IT sees it as the senior admin who "just stopped answering the phone."

How AlertMonitor Solves This: Make Every Page Worth the Interruption

AlertMonitor was designed around a blunt insight: alert fatigue isn't a volume problem — it's a signal quality problem. You don't fix it by turning alerts off. You fix it by making every alert carry enough context to act on.

• Full context on every alert. Each alert carries the device, the client, what changed, and what healthy looks like. A 2am push notification already says: disk on CLIENT-A SQL01 crossed 90%, baseline is 62%, growth accelerated after the backup window, service status normal. Triage starts before you open a laptop.
• Smart deduplication collapses storms into incidents. The 40-endpoint switch flap becomes one incident — "core switch down, 40 devices affected" — with one page instead of forty. Child events are logged against the incident for the post-mortem, not sprayed at whoever is on call.
• Escalation policies with teeth. Multi-level on-call routing is configurable per client and per severity. A critical alert unacknowledged for 15 minutes escalates to the secondary. Another 15 minutes and the manager gets it — via SMS and phone call, not another email into the void. Every acknowledgement is timestamped, so "who saw what and when" is a report, not an argument.
• Maintenance windows actually suppress. Schedule the patch window once; AlertMonitor holds non-critical alerts during it and summarizes everything suppressed afterward. No more 2am pages caused by your own patching schedule — and no more "temporarily disabled" checks that never come back.
• One platform, one timeline. Because monitoring, RMM, helpdesk, network topology, and patch management share a data model, an alert links directly to the device's ticket history, patch state, and position on the topology map. The old workflow — see the alert in the monitoring tool, open a ticket in the helpdesk, remote in through the RMM, cross-check the map, correlate by hand — becomes one screen with one timeline. For an MSP running a NOC across 30 clients, that's the difference between twelve tabs in five tools and a single pane of glass.

Teams that make this switch report the same pattern: overnight pages drop sharply because noise is suppressed and deduplicated, and the pages that do fire get answered fast — because on-call staff have relearned that a page means something.

Practical Steps You Can Take This Week

1. Count your noise. Pull the last 30 days of alerts from whatever you run today. Find the top 10 noisiest checks. Fix the root cause or retire the check. Most teams find a couple dozen checks generate the overwhelming majority of their volume.

2. Give every critical alert an owner and an acknowledgement SLA. If a critical alert has no escalation path behind it, it isn't an alert — it's a log entry with ambitions.

3. Automate the checks that shouldn't be pages. Routine verification — disk headroom, critical services, patch currency — should run on a schedule and only escalate when automation can't resolve it. Here's a disk-capacity sweep across your Windows fleet, outputting only what's actually wrong:

$servers = Get-Content "C:\IT\servers.txt"
$freeThresholdPct = 15

foreach ($srv in $servers) {
    Get-CimInstance -ComputerName $srv -ClassName Win32_LogicalDisk -Filter "DriveType=3" |
        ForEach-Object {
            $freePct = [math]::Round(($_.FreeSpace / $_.Size) * 100, 1)
            if ($freePct -lt $freeThresholdPct) {
                [PSCustomObject]@{
                    Server    = $srv
                    Drive     = $_.DeviceID
                    FreeGB    = [math]::Round($_.FreeSpace / 1GB, 1)
                    FreePct   = $freePct
                    CheckedAt = Get-Date -Format "yyyy-MM-dd HH:mm"
                }
            }
        }
}

4. Self-heal the boring outages. The classic 2am page — a critical service stopped — is usually a restart command wearing an alert costume:

$watched = @("Spooler", "wuauserv", "MSSQLSERVER")
foreach ($name in $watched) {
    $svc = Get-Service -Name $name -ErrorAction SilentlyContinue
    if ($svc -and $svc.Status -ne "Running") {
        Write-Output "$(Get-Date -Format s): $name is $($svc.Status) on $env:COMPUTERNAME — restarting"
        Start-Service -Name $name
    }
}

Same pattern on Linux hosts:

#!/bin/bash
# Restart critical services if they died; log every action
for svc in nginx mysql sshd; do
  if ! systemctl is-active --quiet "$svc"; then
    echo "$(date '+%F %T'): $svc down on $(hostname) — restarting" >> /var/log/selfheal.log
    systemctl restart "$svc"
  fi
done

In AlertMonitor, these become automated remediation policies attached directly to the monitor: detect, attempt the fix, and only escalate to a human if the fix fails. The 2am page becomes a 7am summary of what healed itself.

5. Treat patch currency as a health signal, not a spreadsheet chore. Systems missing updates are systems carrying pending reboots and rising risk. A quick compliance snapshot per host:

Get-HotFix |
    Sort-Object InstalledOn -Descending |
    Select-Object -First 10 @{N='Server';E={$env:COMPUTERNAME}}, HotFixID, InstalledOn, Description

6. Test the pager. Once a quarter, fire a synthetic critical alert at 2am on purpose and time how long it takes for a human to acknowledge. If the answer is "nobody noticed until morning," you have just discovered your CenterPoint moment before it happens — while it still only costs you a lesson instead of a headline.

The breaches that make the news and the file servers that fill up on Saturday night share one root cause: signals that existed but never became action. Fix the signal path — context, deduplication, real escalation, honest suppression — and both stories change. That's the work AlertMonitor was built for.

Related Resources

AlertMonitor Alert Management & On-Call Operations AlertMonitor Platform Overview Book a Demo Alert Management & On-Call Operations Resources