Resolution, restore first and drafted fixes
How 3AM gets a service back, not just diagnosed. Which fixes may run without asking, trying the next fix, restoring service when nothing is proven, the hand-over, and fixes 3AM drafts itself.
3AM is judged by one thing: service restored, and nothing made worse. Safe moves can run as soon as the evidence justifies them, anything that could break something waits for people, and a fix that didn't work is followed by the next one.
Which fixes are safe
Every fix is sorted into one of three kinds:
| Kind | What happens |
|---|---|
| Safe | May run without asking, on services at L2 |
| Needs approval | Waits for your approvers |
| Forbidden | Never runs: 3AM is halted, your policy denies it, or no connector can do it |
A fix counts as safe only if every one of these is true. If any isn't, the approver is told which.
- It's on the short list of safe actions: restarting a Deployment or one of its pods, scaling a Deployment up, stopping a stuck Argo CD sync, or a traffic shift you've declared reversible. Your policy can add to the list, or take away.
- It's low or medium risk, can be undone (or a controller recreates what it replaces), and destroys nothing.
- It takes no free-form statement, query, command or script.
- Its target is stateless: never a StatefulSet, a database, or anything you've marked stateful.
- A scale-up at most doubles the replicas, up to 20.
- The cause is proven, or the suspect is strong and the impact high.
- It's within the limits: at most 2 such fixes per incident, 2 per service and 6 in total each hour, and not the same fix twice within 30 minutes.
Safe fixes still pass your policy, the rehearsal, and verification afterwards.
Autonomy L2
At L2, safe fixes run without asking. Everything else still needs approval. Raising a service to L2 needs two approvers; lowering it applies at once.
Services at shadow and L1 still record what L2 would have run, so the Digest can show whether a service has earned L2 before anyone raises it.
Mitigate first
At L2, when nothing is proven yet but a strong suspect's own check passed and the impact is high, 3AM can run the first safe fix right away, and keep investigating. If the alerts stop, the incident closes as mitigated first, with the suspected cause flagged as still to confirm.
When a fix doesn't work
3AM tries up to three fixes in turn, ordered by impact, how likely each cause is, how likely the fix is to work, and how risky it is. Each goes through the same gates. After each one, 3AM re-checks the cause. If it cleared, it stops. If it didn't, it moves to the next fix. A person rejecting a fix stops it altogether.
Restore first
When nothing is proven, 3AM still looks for the one move that would most likely restore service:
| Move | When | If the alerts don't stop |
|---|---|---|
| Undo the one change | Exactly one thing changed in the 30 minutes before the first alert: a schema, a setting, some reference rows, a deployed image, an Argo CD revision | 3AM undoes its own undo where the reverse is exact, and hands over |
| Start what stopped | Nothing changed, and the application the alerts are about isn't running | Nothing to take back |
| Run your declared failover | Nothing changed, nothing stopped, and exactly one database the service uses is down | 3AM never fails back by itself |
These rules always hold:
- Exactly one change, or nothing. With two candidates, 3AM doesn't guess: both go to the people on call.
- Nothing running is restarted, and 3AM never works out a failover you haven't declared.
- Restore moves only happen with drafted fixes turned on (below). They need two approvers by default, and the request says plainly: Restore move (cause not proven).
- Just before running, 3AM reads the target again. If it has changed, the move isn't run.
With drafted fixes off, the move is not proposed. Instead, the exact command and how to reverse it lead the hand-over.
The hand-over
If no cause is proven 90 seconds after the first alert, the people on call get the triage while the investigation carries on: what's affected and since when, what changed just before, what's abnormal now, what was ruled out, and what couldn't be checked. Services in shadow tell no one.
Drafted fixes
Sometimes 3AM proves a cause but no pack has a fix for it. What happens then is up to you, with one setting for the whole install:
- Advise (the default): 3AM gives the people on call the cause, the evidence and what to do.
- Draft: 3AM drafts a fix and asks for approval, like any other.
Switching to draft needs the same approvers as raising autonomy. Switching back applies at once.
Where a drafted fix comes from. First, 3AM tries to undo the proven change exactly, from the value it recorded before: re-adding a dropped column with its type, putting a setting back, restoring removed rows, rolling a Deployment back. If the old value can't be reproduced exactly, it refuses. Failing that, the investigator fills in one of your connectors' declared actions, with every value taken word for word from the evidence. It's never allowed a free-form statement or command.
The extra gates:
- Only for a proven cause.
- Two approvers by default (
drafted_fixes.quorum, never fewer than one). The request is labelled Drafted fix, with what it undoes, how to reverse it, any caveats, and a warning if a recorded deploy might explain the change. - Just before it runs, the change must still be there and the cause still confirmed.
- It counts as working only if the cause cleared and, for an undo, the old value is back. Otherwise it's handed to a person with the reverse ready. Nothing is undone automatically.
- A rejected or failed drafted fix isn't tried again on that service for 30 minutes.
In the audit log
decision.classified, action.autonomous, resolution.state, restore.suggested, action.reversed,
handover.sent, fix.drafted and fix.draft_refused record each step (Audit events).