The investigation
How 3AM investigates an incident, stage by stage, the rules for calling a cause proven, and what the incident profile records.
Every incident gets the same investigation. It goes stage by stage, from the cheapest checks to the most thorough, and stops as soon as a cause is proven. Every stage only reads your systems. Nothing is changed until a cause is proven and someone approves the fix.
Each incident has a time budget, four minutes by default. If its alerts stop while it's being investigated, it closes as cleared and nothing is done. You can watch every stage live on the incident's page.
1. Known causes
3AM runs the checks of every cause its packs know for these alerts, all at once. A check is a query you could run by hand: PromQL, SQL, a Kubernetes read.
2. Look wider
When no pack proves a cause, 3AM gathers evidence from every direction at once:
- What changed. 3AM keeps a record of how your systems are set up: schemas, settings, permission tables, deployed images. A change shortly before the incident is a strong lead, even for a failure no pack knows.
- What's abnormal. Error rates and latency against an hour ago, restarts, memory kills, targets down, across the service and everything it depends on.
- What the errors say. Error messages from logs and databases, grouped, with the new ones marked.
- The alert's own questions. Any read action the alert's labels can fill in: a pod's logs, a deployment's health, a host's status. Evidence lists what each source reads.
3. Weigh the leads
Every lead becomes a possible cause with a likelihood. How likely depends on how often that kind of cause has been right on your estate before, whether it started before the symptoms, how close it is to the alerting service, and whether separate methods point at the same thing. System 1 scores how well each one fits all the evidence.
3AM then runs the read-only test that best tells the top causes apart, updates the likelihoods, and repeats. The likelihoods decide what to test and show first. They never decide what's acted on.
4. Watch
Some failures take a minute to show themselves. 3AM checks again every 30 seconds for three minutes as more alerts arrive and symptoms build.
5. The investigator
If nothing is proven yet, a language model digs in through your connectors' read-only actions. Every idea it has must come with a check that 3AM runs itself (Models).
What "proven" means
| The cause came from | It's proven when |
|---|---|
| A pack | Its required checks pass on live data, and they didn't already pass half an hour before the first alert |
| A change | It's among the leading causes, it's still in place when checked again, an independent line of evidence agrees, and System 1 agrees the change explains the incident |
| The investigator | Its check passes twice, System 1 agrees, and an independent line of evidence agrees |
| Another incident | The service depends on one whose incident is proven. This one is linked to it, and gets no second fix |
An independent line of evidence comes from the symptoms (alerts, metrics, errors), found by methods that never saw the claim. The investigator agreeing with a change doesn't count: it was shown the change.
A symptom is never a root cause. "The host became unreachable" is a lead, so 3AM keeps looking for why.
The incident profile
Every investigation ends with a profile, shown on the incident's page:
- How sure 3AM is: proven, linked, suspected, disputed, unproven or cleared.
- The chain from root cause to symptom, with the evidence for each link.
- What was ruled out, what was already true before the incident, what was inconclusive, and what couldn't be checked here, with why.
- Everything it tried: each method, what it found and how long it took.
- Runbook excerpts about the proven cause, each checked by System 1.
Evidence is written as plain sentences, such as Check rate_403 on prometheus returned 0.316 (expected above 0.02), so people and System 1 can both read it.
Options
| Option | What it does |
|---|---|
Causal discovery (--causal) | Runs 3am-causal beside 3AM, which finds cause and effect between the metrics that shifted together. Its links become extra evidence |
Tree search (--investigator mcts) | The investigator explores several lines of reads at once instead of one step at a time. Same tools, same rules. Try it on replays of your own incidents before switching |