Over eighteen months we instrumented production estates across banking, healthcare, manufacturing, telecom, retail, and logistics, and captured every alert that reached a NOC queue. The corpus came to 2.4 million alerts. The finding that shaped our entire product direction was not that AI is clever — it is that enterprise IT failure is deeply, almost boringly, repetitive.
The long tail is much shorter than people assume
When alerts are normalised — stripping hostnames, timestamps, and instance identifiers, then clustering on the residual signature — 2.4 million alerts collapse to approximately 180 distinct signatures. The top 25 signatures account for just over 61% of total alert volume. The top 60 account for 82%.
This matters because an incident class that recurs hundreds of times a month has something a novel incident does not: a resolution history. Every previous occurrence carries the diagnostic path an engineer walked, the command they ran, and whether it worked. That is training data in the most literal sense.
The eight failure classes
Nearly all L1 volume — and a surprising share of L2 — falls into eight classes:
- Capacity exhaustion — filesystem, tablespace, log volume, inode pressure.
- Memory pressure — leaking JVM heaps, cache growth, OOM-killer events.
- Service state — a process, container, or Windows service that stopped and needs a controlled restart.
- Session and lock contention — Oracle blocking sessions, deadlocks, long-running queries holding row locks.
- Certificate and credential expiry — TLS certs, service accounts, API keys, Kerberos tickets.
- Queue and backlog growth — message brokers, batch queues, replication lag.
- Synchronisation drift — cluster members out of sync, config drift, failed replication.
- Connectivity — link flaps, DNS resolution failures, firewall path changes.
Each class has a small number of safe, well-understood, reversible remediations. Clearing archive logs after verifying a backup completed. Restarting a stateless service after draining connections. Killing a blocking session after confirming it is not a committed transaction mid-flight. These are not creative acts. They are runbooks that engineers execute at 2am while half asleep — which is precisely why they are executed inconsistently.
Where the 70% figure actually comes from
We classified each of the 2.4 million alerts against three tests: is the signature known, is a deterministic remediation available, and is that remediation reversible without data loss? An alert had to pass all three to be counted as autonomously resolvable.
69.4% of alerts passed all three tests. Roughly 18% failed on reversibility — the fix exists but carries irreversible risk, so it belongs behind human approval. The remaining 12.6% were genuinely novel: new failure modes, cascading multi-system events, or symptoms with no established diagnostic path.
Why this does not eliminate engineers
The 12.6% novel bucket is where engineering value actually lives, and it is where experienced engineers are systematically starved of time. In the estates we measured, senior engineers spent between 40% and 55% of their week on incidents drawn entirely from the top 25 signatures. Automating that band does not reduce headcount requirements; it redirects the most expensive people in the organisation towards the problems that need them.
How to validate this on your own estate
You do not need to take our numbers on faith, and you should not. Run your own alert corpus through the same three tests:
- Export 90 days of alerts from your monitoring platform and normalise out host, time, and instance identifiers.
- Cluster on the residual signature and rank by frequency. Stop at the point where the cumulative curve reaches 80%.
- For each of those signatures, ask the owning engineer: is there a standard runbook, and is it reversible?
- The proportion answering yes to both is your realistic autonomy ceiling — before any AI is involved.
Most estates we have run this exercise with land between 60% and 78%. If your number is materially lower, it usually indicates alert configuration problems rather than a genuinely unusual environment — and fixing those is worth more than any automation layer you could buy.
“Automation earns trust by being boring. The first hundred actions should be the ones nobody would have thought twice about.”
Arjun Kumar, Chief AI Officer