Every operations leader knows alert fatigue is bad. Very few have costed it. This is an attempt to put defensible numbers against something usually discussed in adjectives.
The baseline
Across the estates we have measured, a NOC covering 400 to 800 managed units receives between 1,000 and 1,500 alerts per day. Approximately 70% require no action: duplicates of an already-known condition, transient threshold breaches that self-resolve, or alerts on assets in a known maintenance state.
The direct cost
Take the midpoint: 1,250 alerts per day, 875 of them actionless. Median triage time for an alert that turns out to need nothing — open, read, correlate, confirm, close — is 6.4 minutes when measured from ticket timestamps rather than self-report.
That is roughly 3.9 full-time engineers whose entire year is spent confirming that nothing was wrong. At Indian metro L2 salary bands with benefits and overhead loaded, that is a meaningful line item — but it is not the expensive part.
The indirect cost is the one that hurts
The expensive failure mode is the genuine incident that arrives in the middle of a storm and gets treated with the same reflexive dismissal as the 400 alerts before it. We looked at post-incident reviews for major incidents across 14 estates. In 31% of them, the first signal had been raised and acknowledged more than 20 minutes before anyone recognised its significance — and in the majority of those cases, alert volume in the preceding hour was above the estate's 90th percentile.
Fatigue is measurable within 90 minutes
Median acknowledgement time degrades measurably during sustained alert volume. In the first 30 minutes of a storm, median ack time holds near baseline. By 90 minutes it has typically risen 2.5 to 4x, and the rate of alerts closed as "duplicate" without investigation roughly triples. Engineers are not being careless; they are triaging under a load no human sustains.
Why the usual fix makes it worse
The standard response is threshold tuning and suppression rules. This reduces alert count, which makes the dashboard look better. It also, in every estate where we have examined the suppression rules, quietly suppresses conditions that mattered — because the rule was written during a storm to stop a specific noise source, and nobody revisited it.
What actually reduces the cost
- Deduplicate on signature, not on text. Identical conditions on 60 hosts are one incident.
- Correlate along dependency edges so downstream symptoms attach to the upstream cause.
- Auto-resolve the classes that are unambiguous and reversible, so they never reach a queue.
- Attach evidence to whatever does reach a human, so triage starts from a diagnosis rather than a red square.
- Review suppression rules quarterly and delete any whose author has left the team.
In estates that have done all five, actionless alerts reaching a human drop by 80 to 90%, and the recovered capacity goes to problem management — which is where the recurrence gets eliminated at source.
“Nobody has ever been promoted for the incident that did not happen. That is exactly why alert fatigue persists.”
Rajesh Nair, VP Sales