Incidents and grouping
One failure, one incident. How 3AM groups the alerts a failure sets off, an incident's lifecycle, and which incident is worked first.
A real failure rarely raises one alert. When a database goes read-only, the same service raises knock-on alerts while it's fixed and recovers, the services that depend on it start failing, and the alert may flap on its way out. 3AM groups all of that into one incident, with one ticket and one message thread.
Inside an incident, each investigation of one service's alerts is an episode. The incident is what people see and what gets resolved.
The lifecycle
| State | What it means |
|---|---|
| Open | Its alerts are being investigated |
| Mitigating | A fix is waiting for approval, or running |
| Monitoring | The fix is verified, or every alert stopped. New alerts now join this incident |
| Resolved | 30 quiet minutes after monitoring began |
How alerts are grouped
A new alert joins a live incident when it's clearly part of the same failure:
| The alert | Joins when |
|---|---|
| Is one of the incident's own alerts, firing again | Always, even up to 30 minutes after it resolved: it reopens |
| Names what 3AM's fix acted on, such as the restarted workload | Within 30 minutes of the fix |
| Is on a service already in the incident | While the incident is live |
| Is on a service that depends on one in the incident, or that it depends on | Up to three hops away, while the incident is active |
| Names the same host, node or workload | While the incident is active |
Every grouping decision is in the audit log, with its reasons.
An incident only explains new alerts while its cause still holds. Before it takes in an alert, 3AM checks its cause again. A cause that's been fixed never explains the next failure.
What a grouped alert gets
Grouping never means silence. Each grouped alert is still looked at:
- Recovery noise. Alerts right after a fix get up to two minutes to stop by themselves. If they do, they're the tail of the recovery: no investigation, nobody paged.
- The same cause again. The fix didn't hold. A fix is proposed again, with approval, in the same incident.
- A different cause. If it's contributing to this failure, it stays in the incident. Otherwise it's split into a new incident of its own.
- Still firing after the fix. An update goes to the people on call, on the same thread.
Updates on a thread are titled Update to incident …, at most one every 10 minutes per incident.
Which incident goes first
When more incidents arrive than 3AM has slots for (Sizing), it works on:
- causes before their symptoms, and services others depend on before the services that depend on them;
- then the incidents with the most impact, and critical alerts;
- and nothing waits forever: the longer an incident waits, the higher it rises.
One slot is always kept free for a new, unrelated incident.
Settings
| Setting | Default | What it does |
|---|---|---|
--monitor-minutes | 30 | Quiet minutes after a fix before an incident is resolved, from 5 to 120 |
Open incidents survive a restart.