When a Datacenter Doesn't Come Back: Why Your Patch State Is Your Real Recovery Baseline

A cloud region just proved that "it will come back" is not a plan

The Register reported that a Ukrainian drone attack destroyed a datacenter hosting a Yandex cloud region — and Yandex itself acknowledges that putting the region back into operation might not be possible. Not "degraded for hours." Not "restore in progress." A major cloud provider is publicly admitting that an entire region may simply never return.

If you run infrastructure for a living, your first reaction is probably "that's why we have redundancy." But be honest about what your redundancy actually covers: a failed disk, a dead host, maybe a lost rack. It almost certainly doesn't cover an entire facility ceasing to exist. In that scenario, your plan stops being a failover runbook and becomes a rebuild-from-survivors exercise — and one of the first questions you'll face is one most IT teams cannot answer in real time:

What state is everything that survived actually in?

Not "where are the backups." What patch level are the surviving machines at? Which ones have failed updates from the last two patch cycles? Which are carrying pending reboots that will block cluster joins or agent installs in the middle of a recovery? If answering that takes a day of WSUS exports and guesswork, your recovery timeline just doubled before you restored a single workload.

That's the uncomfortable lesson from this story: patch state isn't a compliance checkbox. It's a core input to your disaster recovery plan. And for most teams, the only system that knows it accurately is a patch tool that isn't connected to anything else.

The problem in depth: patch state lives in a silo, and silos fall over during incidents

Most shops run patching through one of three setups: WSUS, SCCM/MECM, or the patch module bolted onto their RMM (NinjaOne, ConnectWise Automate, N-able). All three share the same architectural flaw: patch state is stored in the patch system, and nothing else reads it at the moment it matters.

Watch what happens in the three scenarios every IT team eventually meets:

Scenario 1: The 2am mystery reboot. Patch Tuesday deployment kicks off at 2am. One server reboots and hangs on "Working on updates, 30% complete." Your monitoring tool pages on-call with a red "host down" alert. The on-call tech has no idea a deployment was running — that information lives in the WSUS console or an RMM patch schedule nobody has opened since it was configured. They spend 45 minutes treating a patch reboot as an outage. By 8am, the helpdesk queue has twelve "server is slow" tickets, an incident record, and a postmortem nobody wants to write.

Scenario 2: The failover that stalls. You lose a site and fail over to surviving hosts. Two of them have pending reboots from last month's .NET update; one has a failed cumulative update. The cluster won't cleanly accept the nodes, the management agent won't install until the reboot clears, and your recovery runbook — written assuming clean hosts — is now fiction. You burn 40 minutes in a war room deciding whether to reboot servers mid-incident.

Scenario 3: The compliance question at the worst time. A client security questionnaire, an auditor, or a cyber-insurance renewal asks for current patch compliance evidence. Your only artifact is a WSUS report from three weeks ago that doesn't reflect the 15 servers you migrated since. Now someone is hand-verifying patch levels during an already brutal week.

Why do these gaps exist? Because the tooling grew up in silos:

• WSUS-era architecture: the patch tool reports what it was told to deploy, not what the device actually did at 2am. Successful deployments and healthy reboots are assumed; nobody verifies the machine came back.
• Monitoring doesn't subscribe to patch schedules. Your monitor sees "host down." It has no context that the downtime was planned, so it pages exactly when it shouldn't — and trains your team to ignore alerts, which is how real outages get missed.
• The helpdesk has no event context. Patch failures surface as vague "my PC is slow" tickets instead of being flagged at the deployment layer with the deployment ID attached.
• Recovery planning ignores software state. Failover runbooks treat every surviving server as interchangeable, when in practice a host carrying two months of patch debt is not ready to receive workloads.

The measurable impact: 30–60 minutes of false investigation per patch-driven "outage," ticket floods after every reboot storm, patch debt discovered during incidents instead of before them, SLA reports you can't defend because monitoring, patching, and ticketing data live in three systems — and technicians burning out juggling four consoles to answer one question: is this machine actually okay?

How AlertMonitor solves this: one device record, one source of truth

AlertMonitor was built on the opposite assumption: patch state is not a separate workflow — it's part of the live health of every managed device. Here's what that changes concretely.

1. Real-time patch status on every managed Windows device — in the same console as uptime. Every device record shows what's missing, what failed, and what's pending a reboot, updated continuously from the agent rather than a weekly WSUS sync. When you need to know which surviving hosts are clean, you filter one view. No export, no reconciliation between reports that disagree.

2. The 2am reboot arrives with context, not as a mystery. Because patch deployments and monitoring share the same platform, a reboot during a scheduled patch window correlates with the deployment. The alert your on-call tech receives at 2:14am reads as "expected — patch deployment in progress on SRV-APP-01" — or, critically, "deployment scheduled, reboot expected, host has NOT come back." That second alert is the one that matters, and it's the one siloed tools structurally cannot send.

3. Scheduled, staged deployments with rollback. Push updates to a pilot group first, then stage by department or device group, with rollback when a patch breaks something. That's expected. What's not standard is that every step is visible next to monitoring and ticket data on the same screen — the MSP tech managing 50 clients doesn't need to open WSUS, then the RMM, then the helpdesk to confirm Tuesday's rollout actually completed.

4. Recovery mode becomes a query, not a hunt. When something takes out part of your estate — a datacenter, a SAN, a failing hypervisor host — you filter your fleet by patch status and pending reboots in seconds and know exactly which hosts can receive workloads and which need remediation first. The Yandex story is extreme; the SAN that dies on a Thursday afternoon is not.

Old way vs. AlertMonitor way:

Practical steps: take these today

Step 1 — Find out which servers are sitting on pending reboots right now. This is the state that silently blocks failovers, agent installs, and patch cycles. Run this against your critical server list:

# Find servers carrying a pending reboot from Windows Update or CBS
$servers = Get-Content C:\temp\servers.txt

$report = foreach ($srv in $servers) {
    $pending = Invoke-Command -ComputerName $srv -ScriptBlock {
        $checks = @(
            Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired'
            Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending'
        )
        ($checks | Where-Object { $_ -eq $true }).Count -gt 0
    } -ErrorAction SilentlyContinue
    [PSCustomObject]@{
        Server        = $srv
        PendingReboot = $pending
    }
}

$report | Sort-Object PendingReboot -Descending | Format-Table -AutoSize

Any True in that output is a server that will not cleanly join a cluster or complete a patch cycle until someone reboots it — preferably on your schedule, not mid-incident.

Step 2 — Audit update failures instead of assuming success. WSUS tells you what it pushed. This tells you what Windows actually recorded:

# List failed update installations on a server using PSWindowsUpdate
Install-Module PSWindowsUpdate -Scope CurrentUser -Force
Import-Module PSWindowsUpdate

Get-WUHistory -ComputerName SRV-APP-01 |
    Where-Object { $_.Result -eq 'Failed' } |
    Select-Object Date, Title, Result |
    Sort-Object Date -Descending |
    Format-Table -AutoSize

Run this against your failover targets first. A failed cumulative update on a host you plan to lean on during a recovery is something you want to discover on a calm Tuesday, not at 3am during an incident.

Step 3 — Establish a patch baseline for the whole fleet. When part of your environment disappears, this list tells you how far the survivors have drifted:

# Last successful update install per server — your recovery baseline
$servers = Get-Content C:\temp\servers.txt

$baseline = foreach ($srv in $servers) {
    $lastPatch = Get-WUHistory -ComputerName $srv |
        Where-Object { $_.Result -eq 'Succeeded' } |
        Select-Object -First 1 -ExpandProperty Date
    [PSCustomObject]@{
        Server    = $srv
        LastPatch = $lastPatch
    }
}

$baseline | Sort-Object LastPatch | Format-Table -AutoSize

Anything older than one patch cycle goes on the remediation list before it goes on the failover list.

Step 4 — For mixed estates, check Linux patch debt too:

# Pending security vs regular updates on Ubuntu
/usr/lib/update-notifier/apt-check --human-readable

Step 5 — Retire the manual version of this workflow. Every script above produces a point-in-time snapshot that is stale the moment you run it. In AlertMonitor, the same data is collected continuously by the agent: create a device group, read the patch status column, schedule a staged deployment with a 2am window, and the platform correlates the resulting reboots with the deployment. Hosts that fail to come back generate contextual alerts and tickets automatically; patches that misbehave can be rolled back. The scripts become a sanity check — not your compliance program.

The takeaway

Nobody in IT controls whether a datacenter gets hit by a drone. What you control is whether your remaining infrastructure sits in a known, healthy, patched state when the unexpected lands — and whether your team learns about a failed patch deployment from a contextual alert at 2:05am or from an angry user at 8:15am. Patch state is recovery readiness. Treat it that way, monitor it continuously, and keep it in the same place as the rest of your operational truth.

Related Resources

AlertMonitor Patch Management & Software Updates AlertMonitor Platform Overview Book a Demo Patch Management & Software Updates Resources