Servicing Mission or Controlled Deorbit? What Hubble's Countdown Teaches IT Teams About Aging Alerting Stacks

NASA is collecting information from industry on two very different endings for the Hubble Space Telescope: mount another servicing mission to keep the 35-year-old observatory alive, or attach a vehicle that steers it into a controlled reentry over open ocean. No decision yet — but the line that should stop every IT operations professional cold is this one: the reentry countdown is ticking.

Because your alerting stack is Hubble.

The Nagios install someone stood up in 2016. The PRTG sensors added one threshold at a time by three different admins, two of whom have since left. The escalation process that is really just an email to a distro list nobody owns anymore. Every unmaintained alert rule is a little more orbital decay. You can keep servicing the veteran system with weekend patches and a rotation of exhausted engineers — or you can plan a controlled deorbit and rebuild on something designed for how IT actually operates today.

The difference between NASA and most IT teams? NASA knows its countdown. Most teams find out their alerting has failed at exactly the wrong moment: the disk fills, the alert never fires (or fires 400 times into a muted channel), and the first monitoring system to detect the outage is an end user calling the help desk.

This article is about recognizing the decay and planning the controlled descent — before the uncontrolled reentry.

The Problem: An Alerting Stack in Decaying Orbit

2:14 a.m., and nobody is looking

Here's the scenario, and you've lived some version of it. A core switch at a client site starts flapping. The monitoring tool fires 200+ device-down alerts because every downstream host looks offline. Meanwhile, fs01's D: volume crossed 90% on Tuesday and has sent the same "Disk space low" email every five minutes since. The on-call tech got seven pages overnight, all noise; by 2 a.m. his phone is on Do Not Disturb. At 3:41 the RAID controller logs a degraded array — a genuinely critical alert — and it lands in the same muted channel as everything else. Nobody sees it until the morning shift opens the dashboard. The file server has now been one failed disk away from data loss for six hours.

If that sounds like exaggeration, pull your own numbers. Industry surveys consistently find the majority of daily alerts are noise: duplicates, low-priority chatter, or flat-out false positives. A mid-size environment generating 800–2,000 alerts a day typically has fewer than 10% that need action. That's not a monitoring system. That's a randomness generator with an SMTP connector.

Why the stack decays

The gaps aren't accidental — they're architectural:

Monitoring tools poll and email. Zabbix, PRTG, Nagios, and SolarWinds were built around periodic checks and static thresholds. They emit events; they don't understand context, baselines, or correlation. Every poll cycle is a new event, so one root cause becomes a hundred alerts.
The RMM, the monitor, and the helpdesk don't share state. ConnectWise Automate, NinjaOne, HaloPSA, ConnectWise Manage, Freshservice — each holds a piece of the truth. The alert clears, the ticket stays open. The ticket is resolved, the alert keeps firing. Nobody can produce an SLA report without exporting three CSVs and hoping the timestamps line up.
**Thresholds are set once and never revisited." Alert at 90% disk" encodes nothing about what healthy looks like for that volume. A 2 TB backup staging drive at 90% is a normal Tuesday. A 100 GB system drive at 90% is a fire. Same alert, wildly different meaning.
Escalation is a distro list, not a schedule. There's no routing by severity or client, no escalation timer, no concept of whose turn it is. Pages go to everyone, which means they effectively go to no one.
Maintenance windows are unknown to the monitoring system. Patch Saturday pages the on-call tech for every planned reboot — which trains the human to ignore the tool. That's how "cry wolf" gets institutionalized.
MSPs multiply all of it by client count. Which client? Which contract? Which SLA tier? Most legacy stacks can't answer any of those questions at page time.

What the decay costs

Detection time collapses into user tickets. When monitoring is noise, the real detector is a frustrated end user — adding 20–45 minutes to every incident before anyone even looks.
MTTR balloons on context gathering. The average critical alert sends a tech through four or five tools — the paging app, the monitor, the RMM for a remote session, the helpdesk for the ticket — before any actual fixing starts.
SLA reporting becomes fiction. You cannot report accurately on response times when the alert data and the ticket data live in different systems that disagree with each other.
Burnout becomes rotation policy. On-call becomes the shift nobody wants. Your best engineer starts browsing job listings after a bad patch week. Turnover follows.
And eventually: the uncontrolled reentry. The missed degraded-array alert. The core switch failure nobody saw coming. The MSP client who churns after one visible outage. NASA at least gets to choose between servicing and deorbit. If you don't choose, the stack chooses for you — usually at 2 a.m.

How AlertMonitor Gets You Out of Decaying Orbit

AlertMonitor was designed around a specific insight: alert fatigue isn't a volume problem — it's a signal quality problem. You don't fix 2,000 daily alerts by hiring more people to read them. You fix it by making every alert worth reading.

Every alert carries full context. Device, client, what changed, and what healthy looks like for that specific asset. "fs01 / D: free space 8.1%, dropped 3.2% in 24h, baseline for this volume is 41% free" is an alert an engineer can act on at 3 a.m. without opening five tabs. "Disk: 92" is not.

Smart deduplication collapses the storms. The disk alert that fires every five minutes becomes one incident with a counter — not 96 notifications. When the core switch drops, the fourteen downstream "device offline" alerts correlate into a single root-cause incident, visible on the network topology map, so the on-call tech sees the switch instead of a scattergun of symptoms.

Multi-level on-call routing. Escalation policies are configurable per client and severity: page the primary on-call, escalate after five minutes unacknowledged, then the secondary, then the service desk lead and account manager. Pages go to the person whose turn it is — with enough context to act.

Maintenance window suppression. Patch Saturday suppresses the forty "server rebooted" alerts automatically. The pages that do come through that night are real — which keeps the on-call tech's phone trustworthy.

One platform, one state. Because monitoring, RMM, helpdesk, topology mapping, and patch management share one database: a critical alert auto-creates a ticket with full context; clearing the alert updates — and can auto-resolve — the ticket; remediation like restarting a service or pushing a patch happens from the alert itself. SLA reporting finally measures reality, because the alert data and the ticket data are the same data.

The workflow, before and after:

Old way: PagerDuty page → log into PRTG to see what fired → log into ConnectWise Automate to remote in → create a ticket in Manage → run checks → update the ticket → close the alert in PRTG. Five tools. Ten minutes of context gathering before any actual work.

AlertMonitor: one alert with context → acknowledge → run remediation from the alert → ticket updated automatically. Minutes, including the fix.

Teams that make this move report mean time to acknowledge dropping from tens of minutes to under two, overnight pages falling by double digits per week, and — the metric that actually matters — on-call staff answering pages again, because pages mean something.

Practical Steps: Plan Your Controlled Deorbit This Week

You don't need a NASA-scale program review. Start here.

1. Audit the noise

Export the last 30 days of alerts from whatever tool you run today. Find the top 20 noisiest rules. Retire the ones that have never once indicated a real problem; fix thresholds on the rest. This exercise alone typically removes 40–60% of daily volume.

2. Encode what healthy looks like — before you alert on it

Don't set thresholds from vendor defaults. Baseline first. Here's a quick disk baseline across your Windows servers:

PowerShell

# Baseline disk health across Windows servers
$servers = 'dc01','fs01','sql01','rds01'

Get-CimInstance -ComputerName $servers -ClassName Win32_LogicalDisk -Filter 'DriveType=3' |
    Select-Object @{n='Server';  e={$_.PSComputerName}},
                  @{n='Drive';   e={$_.DeviceID}},
                  @{n='SizeGB';  e={[math]::Round($_.Size/1GB,1)}},
                  @{n='FreeGB';  e={[math]::Round($_.FreeSpace/1GB,1)}},
                  @{n='FreePct'; e={[math]::Round(($_.FreeSpace/$_.Size)*100,1)}} |
    Sort-Object FreePct |
    Format-Table -AutoSize

Volumes with a stable baseline get meaningful thresholds — alert on a 5% drop in 24h, not a static 90%. In AlertMonitor those baselines are part of every alert's context automatically.

3. Fix recurring alerts with remediation, not repetition

If the same alert fires weekly, either automate the fix or make it a real incident. A service watchdog is the classic example:

PowerShell

# Restart a critical service if it stops, and log the action
$service = 'Spooler'
$server  = 'rds01'

$svc = Get-Service -Name $service -ComputerName $server
if ($svc.Status -ne 'Running') {
    Start-Service -InputObject $svc
    "$(Get-Date -Format u) - $service was $($svc.Status) on $server; restart issued." |
        Out-File 'C:\Logs\remediation.log' -Append
}

In AlertMonitor this becomes a remediation action attached to the alert: the platform restarts the service, logs it, updates the ticket, and only pages a human if it keeps failing. The same watchdog pattern on Linux:

Bash / Shell

#!/bin/bash
# Restart nginx if it is not active, log the action
if ! systemctl is-active --quiet nginx; then
    systemctl restart nginx
    echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') - nginx was inactive; restart issued" >> /var/log/remediation.log
fi

And a quick filesystem sweep for your Linux fleet: