Alert Fatigue Is an Architecture Problem: SLO Alerting

Q: Why does alert fatigue come back a few months after an alert cleanup?

Because a cleanup removes rules while leaving the process that creates them untouched. Most alerting estates grow one rule per incident retrospective, each written as a cause-based threshold on an infrastructure metric: CPU over 90%, a pod restart, a queue deeper than a fixed value. Each is individually reasonable and collectively they page for conditions that self-heal or never affected a user. Deleting the noisiest 40% halves the page count for a quarter, then the next incidents add their rules and the count climbs back. The durable fix changes the layer pages come from: page only on symptoms the user can feel, measured as error-budget burn over multiple windows, demote every cause-based rule to a dashboard or an inhibition source, and put a weekly review of no-action pages in place so rules are retired on evidence rather than in periodic purges.

Q: What is multi-window, multi-burn-rate alerting and why are the numbers 14.4, 6 and 1?

It is the alerting design recommended in the Google SRE workbook for services with an SLO and an error budget. Burn rate is the speed at which the budget is being consumed relative to the rate that would exhaust it exactly at the end of the SLO window, so a burn rate of 1 means on track to spend the whole budget. The workbook suggests paging when 2% of a 30-day budget is consumed in one hour, which is a burn rate of 14.4 (30 days times 24 hours times 0.02, divided by 1 hour), paging at lower urgency when 5% is consumed in six hours, a burn rate of 6, and opening a ticket when 10% is consumed in three days, a burn rate of 1. Each long window is paired with a short one, five minutes for the one-hour tier and thirty minutes for the six-hour tier, so the alert fires only when the burn is still happening and resets minutes after recovery instead of continuing to fire while the long window drains. The result is high precision, because every page corresponds to measurable budget loss, and high recall, because the three tiers cover fast, medium and slow burns.

Q: Should infrastructure alerts like CPU, memory and pod restarts page anyone?

Almost never on a service with autoscaling and self-healing, because they are causes rather than symptoms and most of them resolve without a human. A pod restart that the deployment already replaced, a disk at 81% that will be at 82% next week, or a queue that was briefly deep because a batch job ran on schedule are not incidents; they are context. The right treatment is to demote them, not delete them: keep the rule, stop it paging, show it on the dashboard the responder opens, attach it as an annotation on the symptom-based page ("CPU on 3 of 12 pods over 90% for 10 minutes, last deploy 14 minutes ago"), and where a cause explains a cascade, use it as an inhibition source so that one platform-level alert suppresses the dependent service-level pages. The exceptions are causes with no downstream symptom that a person can act on in time, such as a certificate expiring in seven days or a backup job that did not run, which belong in a ticket queue rather than a pager.

Q: How do you migrate thousands of legacy alert rules to SLO-based alerting without losing coverage?

Treat it as a migration with a coverage contract rather than a cleanup. First, inventory every rule with three fields most estates lack: the owning team, when it last fired, and what the responder did about it; adding an "action taken" choice to the acknowledgement flow for one quarter produces that data. Second, define service level indicators for the top user journeys and wire multi-window burn-rate alerts to them. Third, run the new alerts in shadow mode, routed to a channel nobody is paged by, for four to six weeks while the legacy rules keep paging, and for every legacy page record whether an SLO alert would have fired within an acceptable detection time. The gaps are the real findings, usually a journey with no indicator or a failure that produces neither errors nor latency. Only then demote causes to dashboards and inhibition, delete unowned rules that have not fired in a year, and switch paging to the SLO alerts. Deleting rules first is how a team discovers which ones mattered from a customer.

Q: How do alert grouping and inhibition reduce pages during a cascade?

Grouping batches alerts of a similar nature into one notification, and inhibition suppresses a set of alerts while a related alert is already firing; Prometheus Alertmanager implements both and Grafana alerting follows the same model with a default group-wait timer of 30 seconds. Configured well, a database failure that would otherwise page forty dependent services produces one page for the database owner, with the dependent alerts attached as context rather than delivered as separate pages, and a burst of alerts within one service collapses into a single notification with the constituent alerts listed. The design decisions are the grouping key, which should be the service and an incident key rather than the alert name, the group wait, which must be long enough to collect the burst, and the inhibition rules, which should flow from the platform layer downwards and therefore require the dependency edges to exist in a service catalogue. Cell-based architectures make this tractable because the grouping key can be the cell.

Q: What are the limits of burn-rate alerting for low-traffic services?

The service level indicator needs enough events for the ratio to be meaningful over the short window. A service that receives 20 requests an hour spends 2% of a monthly budget on a single failed request, so a five-minute window flaps on individual errors and precision collapses. The options are to aggregate several low-traffic services into one SLO so the denominator is larger, to lengthen the windows and accept slower detection for services where that is acceptable, or to drive the indicator from synthetic probes sent at a fixed rate so the denominator is stable regardless of user traffic. A second limit is data continuity: a three-day window needs three days of data after any change to the indicator definition, so an SLO migration produces a quiet period that must not be mistaken for health. A third is that burn-rate alerts only catch what the indicator measures; a request that returns 200 with an empty body in eight seconds burns no availability budget unless the indicator includes a content or latency check.

Q: Should anomaly detection or machine learning be used to decide what pages the on-call?

Not as a paging source. Anomaly detection has a legitimate place in finding things nobody thought to alert on, surfaced in a dashboard or a daily digest for a human to triage. As a pager it recreates the cause-based problem with worse explainability: the responder cannot tell from the page whether the model observed a real change or drifted, cannot reason about the threshold, and learns within weeks to acknowledge without reading. Models that learn from production data can also normalise a persistent fault as baseline and go quiet on it. The precision of a page must be something an engineer can inspect and argue with, which is what an error-budget burn rate offers and an anomaly score does not. Where teams have had success with AI in operations it has been downstream of a real page: summarising the incident, correlating the attached causes, drafting the timeline, or proposing a runbook step for a human to approve.

Q: How should on-call load itself be measured and managed?

As a reliability number with a budget, on the same dashboard as latency and availability. The four figures that describe the human experience of the alerting system are pages per on-call shift, the percentage of pages that required no action, the time from page to acknowledgement, and the number of incidents discovered by customers before any alert fired. The first two are the precision of the system and the last is its recall; a no-action rate above roughly one in five means the pager is training people to ignore it, and customer-discovered incidents mean the indicators are missing a journey. Mature teams set a page budget per shift, review every no-action page and every missed incident weekly, feed the resulting rule and indicator changes through pull requests with tests, and treat a breach of the on-call budget as an incident in its own right, because it is one: it predicts attrition of the engineers who know the system best.