How 3AM works

3AM runs inside your network, next to your monitoring. When an alert fires, it gathers evidence from your live systems, proves the cause, and proposes a fix. With your approval, it makes the change and checks that it worked. Everything it does is written to a signed audit log first.

Here is one incident from start to finish: the core banking database goes read-only.

1. An alert firesCopy Anchor Link

Alertmanager fires MySQLReadOnly for core-banking-db. 3AM reads every alert source every 15 seconds, or takes alerts pushed by webhook. It opens an incident and waits 20 seconds for related alerts to join it, so one failure is one incident, with one ticket and one message thread, however many alerts it sets off.

2. It works out what's affectedCopy Anchor Link

3AM knows which service the alert belongs to, who owns it, what code it runs and what depends on it. It learns this from your monitoring labels, CMDB, clusters and repositories, and you can correct it on the Estate page.

3. It proves the causeCopy Anchor Link

A pack lists the known causes of an alert, each with read-only checks: a PromQL query, a SQL query, a Kubernetes read. 3AM runs every candidate's checks at once:

confirmed   db_read_only               The primary database is in read_only mode
refuted     db_connection_exhaustion   A client is leaking database connections
refuted     db_unreachable             The database is down or unreachable

A cause counts as proven only when its checks pass on live data, and it wasn't already true while things were healthy. A check that returns no data proves nothing.

When no pack fits, 3AM looks wider: what changed just before (a schema, a setting, a deployed image), which metrics moved, which errors are new. Outside packs, a cause needs two independent lines of evidence. How 3AM finds the cause

4. It rehearses the fix and asksCopy Anchor Link

The fix for this cause is SET GLOBAL read_only = OFF. 3AM rehearses it first (SQL in a transaction that is rolled back; Kubernetes as a server-side dry run), then asks in your approval channel:

Fix on core-banking-db: SET GLOBAL read_only = OFF
Proven: the primary database is read-only.
Evidence: read_only=ON, super_read_only=ON
Reversible: yes.

Whether it asks, and who, depends on the service's autonomy:

Raising a service's autonomy needs two people to approve. Safety rails

5. It fixes and verifiesCopy Anchor Link

After approval, 3AM makes the change and re-runs the cause's checks until they no longer confirm it. Only then is the incident mitigated. If the fix didn't help, it tries the next candidate, each approved in turn, before handing over.

When nothing is provenCopy Anchor Link

3AM never acts on a hunch. Within 90 seconds it hands the incident to the people on call, with what's affected, what changed, and what it already ruled out. If exactly one thing changed just before the failure, such as a dropped column or a new image, it says how to undo it, or, with drafted fixes turned on, asks for approval to undo it (Restore first).

Where the models fitCopy Anchor Link

The models never act. Checks prove causes, and fixes come from packs and your own connectors.

• System 1 is a small model that checks each cause, fix and sentence against the evidence, and can veto a fix the evidence doesn't support.
• The investigator, Llama 3.3 70B by default, digs through read-only actions when the checks leave an incident unproven. Its ideas count only once 3AM's own checks prove them.

Models

It learns your estateCopy Anchor Link

Every closed incident teaches 3AM something: which fixes worked and what your team says happened. Causes that keep being right rank higher; causes that keep being wrong stop proposing fixes. Learning

NextCopy Anchor Link

Check the requirements, then follow the Quickstart.