179 Records, Zero Checks: What the Met Police Auror Incident Teaches IT Teams About Automation Guardrails

The Metropolitan Police Service just took a public dressing-down from the UK data watchdog because officers uploaded details of 179 people to Auror — a commercial shoplifting-reporting platform — without the required privacy checks and without corporate approval. No breach. No malware. No attacker. Just a useful tool adopted outside any governed process: no impact assessment, no sign-off, no oversight — until a regulator started asking questions nobody could answer.

Now read that again with one word swapped: automation. Because most IT teams run their scripts, scheduled tasks, and RMM remediation exactly the way the Met ran Auror. Well-meaning people, useful tooling, zero governance — until something breaks, or an auditor asks who approved it, and the honest answer is "nobody, it just works."

Reactive IT is what happens when that answer is acceptable. Proactive IT is what happens when it isn't. This post is about closing that gap: runbooks that heal problems before they ever page a human, and canary validation that stops a bad script from ever touching your full fleet.

Every Environment Has a "Dave Script"

You know the one. Dave wrote a cleanup script in 2021, two weeks before he left. It deletes files older than 90 days from a temp path, and it's been running via Task Scheduler on four file servers ever since. It works. Nobody has reviewed it. Nobody would find it in an audit, because it isn't in any runbook, ticket, or change record — it's just... there.

Multiply that across a real environment and the inventory gets ugly fast: cron jobs on Linux boxes inherited from a hosting migration, GPO-deployed startup scripts, an RMM auto-fix someone enabled for one urgent ticket that now runs against every client, a vendor PowerShell module that quietly registered its own scheduled task. None of it is malicious. All of it is unaudited, unmonitored, and unowned — the operational equivalent of 179 records uploaded to a platform nobody vetted.

The Failure Mode: Silent, Fleet-Wide, and Discovered Too Late

Scenario 1: The cleanup script that eats backups. Dave's script pattern-matches a path. Eighteen months later, another admin points a backup staging folder at a similar path. The script starts deleting backup files every night. No alert fires, because the script "succeeds." The problem surfaces during a real restore request — the worst possible moment. Weeks of backups, gone, discovered mid-incident.

Scenario 2: The Friday 4:45 PM fleet push. An MSP tech writes a .NET runtime update script and pushes it to 3,200 endpoints across 42 clients because a security advisory dropped. One client runs a legacy line-of-business app that breaks on the new runtime. Monday morning: 60 tickets, one furious client, two techs pulled off project work for a week, and an SLA report that has to be explained in a QBR. Root cause: "push to all" was one click, and there was no canary step anywhere in the workflow.

Scenario 3: The maintenance task that fails for three weeks. A log-rotation scheduled task on an RDS server runs under a service account. The account password expires. The task fails silently every night for three weeks while the disk fills. The monitoring stack checks ping, CPU, and memory — not "did the maintenance task actually succeed." Saturday, 11:40 PM: disk hits 100%, the app dies, on-call gets paged, and the fix is 45 minutes of manual log surgery at midnight. The tooling saw none of it coming, because the automation itself wasn't monitored.

Scenario 4: The audit question nobody can answer. An auditor — or a client's security reviewer, if you're an MSP — asks: "Show me every automated action that touched this server in the last quarter, and who approved each one." The RMM has script history. The monitoring tool has alerts. The helpdesk has tickets. None of them are linked. Assembling the answer takes days, and parts of it simply don't exist. That's the Met's exact position: not "we did something bad," but "we can't show you the checks, because there weren't any."

Why These Gaps Exist

• RMM platforms optimize for speed, not safety. Fleet-wide script execution is a right-click. Approval workflows exist in some tools; under time pressure they're a checkbox nobody reads, and nothing in the pipeline says "prove this works on five machines before it touches five thousand."
• Monitoring and automation don't share a brain. Standalone monitoring detects conditions but can't act — or acts through fragile webhook duct tape. It doesn't know a runbook ran, whether it succeeded, or whether the condition it fired on actually cleared.
• The helpdesk is blind to automation. When a script changes an endpoint, no ticket is created and no record is linked. Six months later, nobody can correlate "what changed" with "what broke."
• Task Scheduler and cron are invisible. The most widely deployed automation platform in the world sits completely outside most monitoring stacks. Tasks fail silently because nothing watches last-run results.
• It's all siloed. Four or five tools, four or five audit trails, zero shared context. The gaps between tools are exactly where shadow automation lives — and exactly where incidents are born.

The business impact compounds: longer outages (incidents caused by unvalidated change take longer to diagnose because nobody suspects the automation), inflated ticket volume (Monday ticket storms), burned-out techs (2 AM pages for problems a 30-line runbook should have prevented), and audit exposure that turns a routine review into a compliance incident.

How AlertMonitor Closes the Loop — With Guardrails

AlertMonitor treats automation the way the ICO clearly wishes the Met had treated Auror: validate first, approve deliberately, monitor continuously, log everything.

Runbooks attached to alert conditions. When an alert fires — disk above 90%, a service stopped, a queue backing up — an attached runbook executes automatically: restart the service, rotate and compress logs, clear temp, trigger a webhook to a downstream system. The human only gets paged if remediation fails or the condition persists after the configured attempts. Every execution is logged with its trigger, output, and before/after state, and auto-linked to a helpdesk ticket. That's the audit trail from Scenario 4, generated as a byproduct of doing the work.

Canary deployment monitoring. New agent version, new remediation script, new patch — AlertMonitor rolls it to a test ring first and watches the outcome: exit codes, service states, whether alert noise appears on the ring. Criteria met, it promotes. Criteria missed, it stops there. The Friday .NET push burns five lab machines' worth of noise instead of 3,200 production endpoints across 42 clients.

Monitoring watches the automation itself. Scheduled tasks and runbooks are monitored entities in AlertMonitor. When Dave's cleanup script starts failing — or starts succeeding in ways it shouldn't — the condition is caught in minutes, not discovered during a failed restore three weeks later.

One loop, one record. Monitoring detects → runbook heals → helpdesk documents → RMM verifies the endpoint's state. Compare the disk-full incident:

Old way: 11:40 PM, disk hits 100%, app dies. An alert emails a distribution list nobody reads at midnight. 7:10 AM, users report it, a tech RDPs in, finds the full disk, cleans IIS logs by hand. Sixty minutes of human time, an hour-plus of user-facing impact, root cause untouched — and it recurs next month.

AlertMonitor: 11:40 PM, the condition fires. The runbook rotates logs and purges old temp files in about 90 seconds. A verification step confirms free space recovered. A ticket is auto-created with the full runbook log attached, and the alert auto-resolves. The tech reviews it with coffee at 8:30 — forty seconds of human time, zero user-facing impact.

That's the difference between proactive IT as a slide in a deck and proactive IT as a Tuesday night nobody remembers.

Practical Steps You Can Take This Week

1. Inventory your shadow automation

You can't govern what you can't see. Enumerate every non-Microsoft scheduled task across your Windows fleet:

$servers = Get-Content .\servers.txt
$report = foreach ($server in $servers) {
    Get-ScheduledTask -CimSession $server -ErrorAction SilentlyContinue |
        Where-Object { $_.TaskPath -notlike '\Microsoft*' } |
        Select-Object @{n='Server';e={$server}},
                      @{n='Task';e=('{0}{1}' -f $_.TaskPath, $_.TaskName)},
                      State,
                      @{n='Author';e={$_.Author}}
}
$report | Export-Csv .\scheduled-task-inventory.csv -NoTypeInformation

And on Linux, sweep user crontabs:

for user in $(cut -f1 -d: /etc/passwd); do
  crontab -l -u "$user" 2>/dev/null | grep -vE '^#|^$' | sed "s/^/$user: /"
done

Then do what the Met didn't: for each item, record what it does, who owns it, and whether it's still needed. Kill the orphans. In AlertMonitor, the survivors become monitored entities — you get alerted the moment a task fails instead of three weeks later during an outage.

2. Guard-rail your destructive scripts

A remediation script should refuse to act until its preconditions are proven, and state its result in an exit code a machine can read:

$drive = Get-PSDrive -Name 'C'
$freePct = [math]::Round(($drive.Free / ($drive.Used + $drive.Free)) * 100, 1)

if ($freePct -ge 15) {
    Write-Output ('Disk has {0}% free. No cleanup needed.' -f $freePct)
    exit 0
}

$freed = 0
foreach ($path in @('C:\Windows\Temp', $env:TEMP)) {
    if (Test-Path $path) {
        $before = (Get-ChildItem $path -Recurse -ErrorAction SilentlyContinue |
                   Measure-Object Length -Sum).Sum
        Get-ChildItem $path -Recurse -ErrorAction SilentlyContinue |
            Where-Object LastWriteTime -lt (Get-Date).AddDays(-7) |
            Remove-Item -Recurse -Force -ErrorAction SilentlyContinue
        $after = (Get-ChildItem $path -Recurse -ErrorAction SilentlyContinue |
                  Measure-Object Length -Sum).Sum
        $freed += ($before - $after)
    }
}
Write-Output ('Freed {0:N0} MB. Free space at start: {1}%.' -f ($freed / 1MB), $freePct)
exit 0

Thresholds, age limits, and idempotent behavior are your script-level impact assessment: documented proof that it acts narrowly and only when needed.

3. Make every remediation verifiable

A restart isn't done when the command returns — it's done when the service is confirmed healthy:

$service = 'Spooler'
Restart-Service -Name $service -Force
Start-Sleep -Seconds 10

$status = (Get-Service -Name $service).Status
if ($status -ne 'Running') {
    Write-Output ('REMEDIATION FAILED: {0} is {1} after restart.' -f $service, $status)
    exit 1
}
Write-Output ('REMEDIATION OK: {0} is {1}.' -f $service, $status)
exit 0

Exit code 0 with a verified state, or exit code 1 and an escalation. That contract is what lets a runbook run unattended at 2 AM with confidence.

4. Canary everything before it touches the fleet

Before promoting any script, agent, or patch: deploy to a ring of five to ten representative endpoints. Define success in machine-checkable terms — exit codes, service states, no new alerts from the ring for a set window. AlertMonitor's canary deployment monitoring watches those signals automatically and gates the fleet rollout on them. Make "did we test it on the ring?" the first question in every incident review involving automation.

5. Attach the loop in AlertMonitor

The workflow takes minutes: define the alert condition (for example, C: free space below 10% for 10 minutes), attach the guard-railed runbook from step 2, set max attempts to 2, and escalate to on-call only if the condition persists after the runbook runs. From that moment, the disk-full 2 AM page stops existing as a category of incident — and every execution lands in the helpdesk with a complete audit trail, ready for the next auditor who asks.

The Met's Auror finding was never really about a shoplifting platform. It was about what happens when capable people run powerful tooling with no checks between intent and impact. Your automation deserves better — and with runbooks, canaries, and unified visibility, better is a configuration, not a project.

Related Resources

AlertMonitor Self-Healing & Proactive IT AlertMonitor Platform Overview Book a Demo Self-Healing & Proactive IT Resources