Safety rails
Shadow mode, per-service autonomy, policy rules, rehearsals, budgets and the halt switch, with the settings for each.
Autonomy, per service
| Level | 3AM | Set by |
|---|---|---|
off | Ignores the service | Anyone, at once |
shadow | Diagnoses and records what it would do. Asks no one, changes nothing, not even a note on a page | Default for every service |
L1 | Asks for approval for each fix, then makes it | Two approvers must agree to raise a service to L1 |
L2 | As L1, but fixes classified safe (a rolling restart of a stateless workload, a bounded scale-up) run without asking | Two approvers, like L1 (L2) |
Lowering a service applies immediately. Raising it sends a request to the autonomy_change chain in the policy:
"default_autonomy": "shadow",
"autonomy_change": { "quorum": 2, "chain": [{ "channel": "slack" }], "expires_s": 3600 }Policy rules
Rules are checked in order and the first match wins. They match on action, risk, reversible, environment,
service, connector or origin (drafted), then refuse the action, require a quorum, limit the hours it may run
in, or say whether L2 may run it unasked (autonomous).
"rules": [
{ "match": { "action": "mysql.start" }, "deny": true, "reason": "database restarts go through the DBA team" },
{ "match": { "risk": "high" }, "quorum": 2, "hours": "07:00-22:00" },
{ "match": { "environment": "prod", "reversible": false }, "quorum": 2 }
]With no rule, low and medium risk need one approver and high risk needs two. When two approvers are required, only high-assurance channels are asked: Slack, Symphony, PagerDuty, Twilio voice. E-mail is never one of them.
Budgets
"budgets": { "per_service_per_hour": 3, "global_per_hour": 10 }Once a budget is used up, further fixes that hour are refused with the reason.
Rehearsal
Every fix runs once in a mode that changes nothing, before anyone is asked:
| Kind | Rehearsal |
|---|---|
| SQL | Inside a transaction that is rolled back. Statements that can't be rolled back are flagged as such |
| Kubernetes | Server-side dryRun=All: the API server and your admission webhooks validate it |
| Silences | Lists the alerts it would silence |
| Reloads | States which component would reload |
A failed rehearsal stops the fix (decision: dry_run_failed) and the incident goes to a person.
Reviewed before anyone is asked
Before a fix is proposed, System 1 reads the evidence and is asked whether the fix addresses the cause. It blocks
the fix when it's at least 70% sure it doesn't (decision: vetoed), and the incident goes to a person. When it's
less sure, the approval request says so. System 1 can block a fix, never approve one (Models).
A cause that has been wrong too often stops proposing fixes: it still diagnoses, and the incident gives its advice to a person (Learning). A fix 3AM learned from your incidents is proposed only after an admin promotes it.
Checked again before running
Approval can take minutes. Just before a fix runs, 3AM checks again: is 3AM halted, is the service still at L1, is
there budget left? If anything changed, the fix is not run (decision: blocked), and the audit log records why.
Verification
After a fix, 3AM re-runs the cause's checks until they no longer confirm it, then closes the incident as mitigated.
If the cause doesn't clear in time, the incident closes as unresolved and goes to a person. An action that claims
success but changed nothing is reported as failed.
Fixes 3AM drafts, and restore moves
Both need fix mode draft (turned on with approval), two approvers each by default, System 1's review of the change
itself, a re-check just before they run, and verification that the fact is back. A restore move that doesn't help is
taken back where the reverse is exact. Resolution.
Halt switch
Refuses every action on every service until someone resumes. Diagnosis continues. See Halt switch.
What 3AM never does
- Act on a service in
shadoworoff. - Take a breaking action without approval: only fixes marked safe run unasked, and only at L2.
- Act on a cause that isn't proven: a suspicion, a disputed cause, or one that was already true before the incident.
- Act without the approvals the policy requires.
- Run anything but a pack's remediation, or an action your connectors declare, with every value taken from the proven evidence.
- Let a model run anything: models only read, through read-only actions.
- Retry a fix someone rejected.
- Guess between two changes, restart something that is running, or work out a failover you didn't declare.
Resolution, restore first and drafted fixes
How 3AM gets a service back, not just diagnosed. Which fixes may run without asking, trying the next fix, restoring service when nothing is proven, the hand-over, and fixes 3AM drafts itself.
Models
The two models 3AM uses, what each is allowed to do, the rules 3AM enforces on them, and what a hosted model sees.