Incidents
Every incident from the first alert to its result, live while 3AM investigates; the diagnosis, the hand-over, what your team did; every decision, outcome and result explained.
Incidents (/incidents, g i) opens with Investigating now: the incidents 3AM is working on, with the step
each is on. Below: every incident, newest first, one row per incident (its related episodes inside). Filter by
kind, service, confidence, decision, outcome and result; search with words or the
query language; export CSV. Keys: j/k move, ↵ opens, n/p page, f filter,
r refresh.
service:core-banking outcome:mitigated
alert:MySQL* OR alert:HighErrorRate
result:handed_over opened<7d
-kind:symptom duration>5mOne failure, one incident
An incident owns every episode of the same failure (one investigation of one service's alerts), one ticket and one message thread, and outlives its fix by a 30-minute monitoring window. Knock-on alerts, the services that depend on it failing, the recovery's tail and the alert flapping all join it. Incidents and grouping.
| State | Meaning |
|---|---|
open | Alerts are being investigated |
mitigating | A fix is waiting for approval or running |
monitoring | The fix is verified, or every alert stopped; new related alerts join this incident |
resolved | Quiet for 30 minutes. Its own alert firing again within 30 minutes reopens it |
While 3AM investigates
The page exists from the first alert and is live: what 3AM is doing now (gathering alerts, checking known causes, looking wider, weighing leads, re-checking, the investigator reading, writing the diagnosis), what it has tried and found so far, the leading causes with their likelihood, and the timeline as it happens. Nothing is done during the investigation.
The hand-over. If nothing is proven 90 seconds after the first alert, the people on call get the triage while the investigation goes on: what is affected and since when, what changed just before, what is abnormal, what's ruled out, what couldn't be checked.
A closed incident
| Part | What it shows |
|---|---|
| Diagnosis | How sure 3AM is (proven, linked, suspected, disputed, unproven, cleared) and the chain: the root cause, what it caused through, and the symptom, each with its evidence sentences and System 1's reading |
| Note and fix | The note for the people on call and who wrote it (the model, or 3AM's own template and why). The fix: its risk, whether it can be undone, the rehearsal, System 1's review, and whether it was a drafted fix or a restore move. Advice when a person must act |
| What did your team do? | We did what 3AM proposed, We followed 3AM's advice, We did something else (say what) or Nothing was needed. Counts towards L1 readiness and teaches 3AM (Learning) |
| Investigation | Every method, what it did, what it found, how long it took |
| Hypotheses, weighed | Every possible cause with its likelihood, where it came from, what corroborates it, and its verdict; the tests 3AM chose and why |
| Root causes checked | Every pack cause: proven, ruled out, open, already true before the incident, or not checked here and why |
| The incident | Its other episodes and the relation that grouped each (same service, dependency, refire, …) |
| Timeline | Every step, newest first; include every check shows each query and its raw result |
Decisions
| Decision | Meaning |
|---|---|
act | The fix was asked for (or, at L2 and safe, run without asking), and made if approved |
shadow | Proven with a fix; in shadow nothing changed, the rehearsal shows what would have run |
restore | Nothing proven, but one move restored service (undo the change just before, start what stopped, the declared failover) |
mitigated_first | At L2, a safe mitigation before proof stopped the alerts; the suspected cause is still to confirm |
ask_human | Proven, but the fix needs a person (advice), System 1 disputes the cause, or nothing could be drafted |
vetoed | System 1 reads the fix as not addressing the cause (p ≥ 0.7) |
same_cause | Another open incident already proved the same failure: linked there, no second fix |
downstream | A service this one depends on has a proven incident |
related | A related episode with nothing of its own proven; explained while the incident's cause holds |
settled | A related episode whose alerts stopped within 2 minutes of its incident's fix: the recovery's tail |
recurrence | One of an incident's own alerts came back after its fix |
abstain | Nothing could be proven; everything tried is handed over |
cleared | The alerts stopped on their own during the investigation |
denied | Policy refused the fix (with the reason) |
dry_run_failed | The rehearsal failed, so nobody was asked |
no_approval_route | No channel can reach the quorum, e.g. two approvers needed and only e-mail set up |
change_blocked | A change record was required and couldn't be opened |
blocked | Approved, but halted, the service lowered, the budget used, or the fix no longer needed before it ran |
Outcomes
| Outcome | Meaning |
|---|---|
mitigated | Verified: the checks show the cause has cleared (for a restore move: the alerts stopped and the fact is back) |
unresolved | The fix ran but the cause is still confirmed; 3AM tried the next candidate or handed over |
action_failed | The fix failed, or reported success without changing anything |
escalated | Handed to people |
explained | A symptom of another proven incident |
rejected | The approver said no |
cleared | Gone on its own |
shadow | Recorded only |
abandoned | 3AM stopped while investigating it (a restart). Nothing after its last event was done |
recurrence means the same cause came back within 30 minutes of an earlier incident on the service clearing. Held:
beyond the licence means the service is over the licence's services limit (Licensing).
API
GET /v1/episodes?group=incident (the list), GET /v1/episodes/<id>[?after_seq=N] (one, with an incremental
timeline), GET /v1/incidents, POST /v1/episodes/<id>/team (what your team did). API.