How 3AM finds the cause
The investigation behind every incident, what "proven" means, and the confidence 3AM reports.
When an alert fires, 3AM investigates it the way a careful engineer would. It checks the known causes first, then looks wider: what changed, what is abnormal, what the errors say. It weighs every lead and tests the most telling one next. It acts only on a cause it has proven against your live systems, and only with your approval.
The investigation
| Step | What 3AM does |
|---|---|
| Known causes | Runs the read-only checks of every cause its packs know for these alerts |
| Looks wider | Checks dependencies, recent deploys and changes, what changed in your systems, what is abnormal in your metrics, what the errors say, the servers and your monitoring itself. All at once |
| Asks what the alert implies | The golden signals against an hour ago, error logs, failing traces, and every read action the alert's labels can fill in (Investigation) |
| Root-cause analysis | Impact and blast radius; System 1 scores every lead against the whole evidence; the investigator proposes a differential; tests run by information gain, twice (RCA engine) |
| Weighs the leads | Every lead becomes a possible cause with a likelihood. 3AM then runs the test that best tells the top ones apart |
| Looks again | Re-checks every 30 seconds for three minutes, as symptoms build up and more alerts arrive |
| Digs in | If nothing is proven yet, a language model investigates through your connectors' read-only actions, and must prove each idea with a check 3AM runs itself (Models) |
| Concludes | Writes the incident's diagnosis: the chain from root cause to symptom, the evidence for each link, and everything it tried |
Each incident has its own time budget (four minutes by default). You can watch an investigation live in the console: the step it's on, what it has tried, and the leading causes (Incidents).
What 3AM looks at, beyond the packs
- What changed. 3AM keeps a record of how your systems are set up: database schemas and settings, small configuration tables such as roles and permissions, server state, deployed images and revisions. It reads them every ten minutes and when an incident opens. A column dropped, a permission removed from a role or a setting flipped shortly before the incident is a lead, even if no pack knows that failure.
- What is abnormal. It scores the metrics of the service and everything it depends on against their last hour,
and finds when each shift began. A rise in
403responses shows up even when total traffic stays flat. - What the errors say. It groups the error messages your database and applications report, and marks the ones that are new since the incident began.
Evidence says what each of these reads, and how customer data is protected.
What "proven" means
A cause is proven only when all of the following hold:
- A deterministic check passes against your live system: a query, a metric, a read you could rerun by hand. Runbooks, past incidents and model output can point the way; they never prove anything.
- It is what changed now. A cause that was already true half an hour before the alert, while everything was healthy, is not this incident's cause (a backlog that has been there for days, a target that's always down). 3AM reports it as already true before the incident and keeps looking.
- Outside a pack, a second line of evidence agrees. A change, or an idea from the model, also has to fit what the symptoms show: the alerts, the abnormal metrics, the errors. A single line of evidence is never enough.
- System 1 doesn't dispute it. A small verification model reads the evidence. If it reads it as contradicting the cause, the incident goes to a person instead.
A change can't be proven as the cause just because something stopped working. "The exporter went down" or "the host became unreachable" is a symptom, so 3AM looks for why.
Confidence
Every incident says how sure 3AM is:
| Confidence | Meaning | What 3AM does |
|---|---|---|
| Proven | The cause passed the checks above | Proposes the fix, or gives the advice when the fix needs a person |
| Linked | A symptom of another incident whose cause is proven, through a dependency checked live | Links to that incident; no second fix |
| Suspected | Pointed at, not proven | Hands it to a person with the evidence. Never acts on a suspicion |
| Disputed | The checks prove it, but System 1 reads the evidence as refuting it | Hands it to a person |
| Unproven | Nothing proven | Hands it over with everything it tried |
| Cleared | The alerts stopped on their own during the investigation | Nothing to fix |
When nothing is proven
3AM never acts on a suspicion. It looks for the one move that restores service (undo the single change just before, start what stopped, the declared failover) and, 90 seconds after the first alert, hands the people on call the triage (Restore first).
When one failure causes many alerts
- The same service: a new alert joins its open incident when the proven cause explains it. It gets its own diagnosis when a pack could prove a different cause, so a second failure is never hidden behind the first.
- Another service: when a service depends on one whose incident is proven, its incident is linked there. You get one fix, not one per symptom. When two pack causes are the same failure seen by two packs, they're treated as one.
- One incident: every alert of the same failure joins one incident, with one ticket and one thread (Incidents and grouping).
- The cause stops holding: an open incident explains new alerts only while its proven cause still holds. 3AM re-checks the cause before attaching anything to it, so a cause that has been fixed never explains the next failure.
After the investigation
What happens next depends on the service's autonomy: shadow records what 3AM would have done; L1 rehearses the fix, asks for approval, makes the change and verifies that the cause has cleared. Then everyone's answers feed back into 3AM (Learning).