The Oldest Protocol in Your Rack Is Still the Smartest Thing in It
Last month, The Register ran a column celebrating TCP/IP — "the wiring of the world" — with a line that should quietly embarrass every vendor in the monitoring space: nothing this old should scale this well, yet here it is. A protocol designed when memory was measured in kilobytes still moves nearly all traffic on Earth. No rewrite, no "TCP 2.0." It works because it was built with ruthless pragmatism: best-effort delivery, intelligence at the endpoints, and — the part that matters to anyone carrying the on-call phone — backoff. When TCP hits congestion, it does not scream louder. It retransmits less, waits longer, and finds a path that works.
Now look at the alerting stack sitting on top of that same network in most IT shops. A core switch port flaps at 2:07 a.m. Within ninety seconds every downstream device times out: twelve servers fail ping checks, agent heartbeats drop, SNMP polls fail, the rack UPS reports communication lost. Three separate tools — your RMM, your standalone monitor, and your ISP's portal dashboard — each fire independently. By 2:10 your phone has buzzed 47 times. Every one is a symptom. Exactly one is the cause. And because your helpdesk isn't connected to any of it, the morning shift learns about the 90-minute outage from an angry voicemail anyway.
If you have lived this, you already understand what the TCP/IP designers knew decades ago: a system that responds to load by generating more load is broken by design. Your monitor paging you on a fixed five-minute drumbeat, regardless of what you have already seen, is the anti-TCP. Alert fatigue is not a volume problem. It is a signal quality problem — and that is precisely the problem AlertMonitor was designed around.
The Problem in Depth: Three Tools, Zero Shared Truth
Walk through what actually happens in a fragmented stack, and be honest if this is your shop. Your RMM (NinjaOne, ConnectWise Automate, Datto RMM) watches agents and patch state. Your standalone monitor (PRTG, SolarWinds, Zabbix) watches network gear over SNMP. Your helpdesk (ConnectWise Manage, Freshservice, HaloPSA) watches tickets. Each has its own threshold engine, its own notification rules, its own definition of "critical" — and no shared state between any of them.
That architecture produces four specific, recognizable failures:
1. Cascading noise with no root-cause correlation. One upstream failure fans out into dozens of downstream "down" alerts. On a 200-endpoint client site, a failed WAN router routinely generates 60–150 individual alerts across your tools. Every single one pages. Your tech wakes up, opens four tabs, and manually reconstructs the dependency tree at 2 a.m. — a topology your monitoring platform already had and didn't use.
2. Alerts with no context. "Disk C: 91% on SRV-APP-03." Which client? Is that drive normally 88% on Sunday nights because of backup staging? Was a maintenance window scheduled? The alert doesn't say. The tech remotes in, finds backup churn that self-cleans, and nobody tunes the threshold. The same false positive fires next Sunday. Forever.
3. No deduplication, no backoff. TCP scales its retransmission interval up exponentially because its designers understood that hammering a degraded path makes everything worse. Most monitoring stacks do the opposite: fixed five-minute re-notification per alert, per tool, until someone acknowledges all 47 of them. The pages multiply precisely when the situation is worst.
4. Data that can't testify together. When the IT manager asks for MTTA on last Tuesday's P1, the answer requires exporting monitoring timestamps from one tool, ticket timestamps from another, and reconciling them in Excel. SLA reporting becomes archaeology. Because alerts never auto-create or enrich tickets, the helpdesk incident record is whatever the end user bothered to type.
The business impact is measurable: the actionable page arrives buried under a storm, so time-to-acknowledge stretches from 90 seconds to 20-plus minutes for the alerts that actually matter; on-call techs burn out and rotate off the roster; the client paying for a 15-minute response SLA gets 40 minutes because the tech was triaging noise. Meanwhile, your best engineer spends their highest-value hours doing manual correlation a machine should do.
The TCP/IP Lesson, Applied to Alerting
The genius of TCP/IP was never features — it was discipline. A handful of rules, applied consistently, that made behavior predictable under stress. Alerting needs the same discipline. Here is how AlertMonitor implements it:
Signal quality over volume. Every AlertMonitor alert carries full context in the payload: the device, the client, what specifically changed, and what healthy looks like for that asset based on its own baseline. The disk alert reads "FS-CLIENT01, drive C: free space fell from a 22% baseline to 9%, growth rate up 4x in three hours, projected full in about five hours." That is actionable without opening a second tab — and the Sunday-night backup churn never pages, because the baseline says it is normal.
Smart deduplication — exponential backoff for humans. When a root cause triggers a storm, AlertMonitor correlates downstream failures into a single root-cause alert with the affected-children list attached. One page, not forty. Re-notification backs off on a configurable curve instead of a fixed drumbeat — TCP retransmission logic, applied to paging.
Escalation that routes, not broadcasts. Multi-level on-call policies: first-line gets a channel notification; if unacknowledged after your configured window it escalates to tier two, then to a phone call — for genuinely critical, actionable events only. P3 informational items wait for business hours. The 2 a.m. phone buzz means something again.
Maintenance windows that suppress, not discard. Planned patching Saturday night? Notifications for scoped assets are suppressed, and AlertMonitor still records every suppressed condition so the post-change review catches anything real. Nothing pages; nothing is lost.
One pipeline from detection to resolution. Because monitoring, RMM, helpdesk, network topology, and patch management share one platform, a critical alert can auto-create a ticket pre-populated with device, client, and diagnostic context; the tech pivots straight into remote management on that endpoint; and if the fix is a patch, patch state is already visible. Detection, triage, action, and the SLA record live in one system — so your MTTA/MTTR reporting is a query, not an Excel archaeology project.
Old way: switch flap → 47 pages across three tools → four tabs → manual correlation → 22 minutes to acknowledge the real problem → ticket created from scratch after the fact. AlertMonitor way: switch flap → one correlated root-cause alert with topology context → auto-ticket → tech acts from the alert. The old flow costs your team hours per incident; the new one is measured in minutes — and the on-call tech is back asleep in four.
Practical Steps You Can Take Today
1. Audit your noise before you tune anything
You cannot fix signal quality you have not measured. Pull the last 30 days of alerts from each tool and count by type. While you're at it, put a real pre-emptive check on the failures that actually page you. The disk-fill alert that wakes everyone at 2 a.m. usually warned you for days — the threshold was just badly placed. Here is a fleet-wide disk check to run right now across your Windows servers:
$servers = @('DC01','FS01','SQL01','RDS01','APP01')
$report = foreach ($srv in $servers) {
Get-CimInstance -ComputerName $srv -ClassName Win32_LogicalDisk -Filter 'DriveType=3' |
Select-Object @{n='Server';e={$srv}},
@{n='Drive';e={$_.DeviceID}},
@{n='FreeGB';e={[math]::Round($_.FreeSpace/1GB,1)}},
@{n='FreePct';e={[math]::Round(($_.FreeSpace/$_.Size)*100,1)}}
}
# Anything under 20% free is a candidate for a trend-based alert, not a static 90% threshold
$report | Where-Object { $_.FreePct -lt 20 } | Sort-Object FreePct
$report | Export-Csv 'C:/Reports/DiskCheck.csv' -NoTypeInformation
Run it, study the FreePct column, and set your AlertMonitor thresholds against trend rather than a single static line. That one change kills most after-hours disk pages.
2. Automate the service-restart pages away
A large share of overnight pages are "service stopped" events a first-line tech resolves with one click. Self-heal the routine ones so the pager only fires for what is genuinely stuck:
$watchlist = @{
'FS01' = @('Spooler','Netlogon')
'SQL01' = @('MSSQLSERVER')
'RDS01' = @('W32Time')
}
foreach ($server in $watchlist.Keys) {
foreach ($svc in $watchlist[$server]) {
$status = Invoke-Command -ComputerName $server -ScriptBlock {
param($name) (Get-Service -Name $name).Status
} -ArgumentList $svc
if ($status -ne 'Running') {
Write-Output "$server : $svc is $status - attempting restart"
Invoke-Command -ComputerName $server -ScriptBlock {
param($name) Start-Service -Name $name
} -ArgumentList $svc
} else {
Write-Output "$server : $svc OK"
}
}
}
In AlertMonitor, attach remediation like this to the alert itself, then set the escalation policy so only a failed auto-remediation reaches a human. Every service restart that never becomes a page is sleep your on-call tech keeps.
3. Same discipline on Linux endpoints
If you run mixed environments, a simple custom check like this feeds cleanly into AlertMonitor as a script-based sensor — exit 1 becomes an alert with full context, exit 0 stays quiet:
#!/bin/bash
# check-disk.sh — exit 1 triggers an alert; keep thresholds trend-aware
THRESHOLD=85
FAIL=0
while read -r use mount; do
pct="${use%\%}"
if [ "$pct" -ge "$THRESHOLD" ]; then
echo "ALERT: $mount is ${pct}% full"
FAIL=1
fi
done < <(df -P | awk '/^\/dev\// {print $5, $6}')
if [ "$FAIL" -eq 1 ]; then
exit 1
fi
echo "All filesystems below ${THRESHOLD}%"
exit 0
4. Rebuild escalation policies around these principles
This is where AlertMonitor replaces the fixed-interval drumbeat:
- Dedup window: correlated alerts inside the window collapse into one root-cause signal with the affected-asset list. The storm becomes one notification.
- Tiered escalation with backoff: level-one channel notification; unacknowledged after ten minutes escalates to tier two; phone call reserved for critical-and-actionable. Retries space out instead of repeating every five minutes.
- Context required: no alert reaches a human without device, client, what changed, and the healthy baseline. If you cannot fill in those four fields, it is not a signal yet.
- Maintenance windows: scope them per client and asset, confirm suppression is logged, and review the log after every change window.
5. Close the loop with the helpdesk — automatically
Set critical alert severities to auto-create tickets in the integrated helpdesk with full alert context attached. One setting, three wins: your on-call tech triages from one screen, the client-facing incident record starts at detection time instead of when the user called, and your SLA reports finally reflect reality because every timestamp comes from the same system.
TCP/IP earned its reputation — nothing this old should scale this well — by refusing to add complexity where discipline would do. Your on-call operation deserves the same treatment. Give every page context, deduplicate the storms, back off instead of drumming, and let the platform correlate what you are currently correlating by hand at 2 a.m. The pager gets quieter. Response gets faster. That is not a compromise; that is the entire point.
Related Resources
AlertMonitor Alert Management & On-Call Operations AlertMonitor Platform Overview Book a Demo Alert Management & On-Call Operations Resources
Is your security operations ready?
Get a free SOC assessment or see how AlertMonitor cuts through alert noise with automated triage.