Approvals

Before 3AM changes anything, someone approves it in a tool they already use. The request carries what they need to decide: the confirmed cause, the evidence, the exact change, the rehearsal's result and whether it can be undone.

Fix on core-banking-db: sql.write SET GLOBAL read_only = OFF (via sql)
Confirmed: db_read_only (The primary database is in read_only / super_read_only mode; every write fails).
Evidence:
  - vars (contains ON): read_only=ON, super_read_only=ON
Dry run: would run: SET GLOBAL read_only = OFF
Reversible: yes.

ChannelsCopy Anchor Link

None of them connects in to 3AM: 3AM reads the answers from the tools. Every decision is recorded with the approver's identity (Slack user, Symphony e-mail, PagerDuty e-mail, phone number).

Who may approveCopy Anchor Link

Each channel takes an approvers list. Answers from anyone else are ignored.

{ "name": "slack", "kind": "slack",
  "settings": { "bot_token": { "file": "/etc/3am/secrets/slack-bot-token" }, "channel": "C07SREAPPR",
                "approvers": ["U02ABCDEF", "anna.k@bank.example"] } }

You can narrow it further per service in the policy:

"services": { "core-banking-db": { "environment": "prod", "approvers": { "slack": ["U02ABCDEF"] } } }

EscalationCopy Anchor Link

approval_chain lists channels in order. after_s is how long to wait before also asking that channel. The first valid decision from any channel already asked counts; the others are closed with a note.

"approval_chain": [
  { "channel": "slack" },
  { "channel": "pagerduty", "after_s": 300 },
  { "channel": "phone", "after_s": 900 }
]

Each channel is a connector's name.

Incidents handed to peopleCopy Anchor Link

When an incident on an L1 service needs a person (the cause is proven but the fix needs judgement, nothing could be proven, or a fix didn't clear the cause), 3AM tells the people on call. It posts the incident note, what to do and the incident id to the first channel of your escalation that can post a message: Slack, Symphony or e-mail. Nothing needs approving in it. To add a link to the incident, give 3AM the console's address as people reach it: ./install.sh --console-url https://3am.bank.internal, or consoleUrl in the Helm chart.

Services in shadow tell no one: shadow changes nothing and asks no one until you raise the service.

Two approversCopy Anchor Link

The same person approving twice counts once. When two approvers are needed, only high-assurance channels are asked. If none is configured, 3AM doesn't fall back to e-mail: the incident is escalated with decision: no_approval_route.

A single reject ends the request.

ExpiryCopy Anchor Link

An unanswered request expires after 30 minutes (60 for autonomy changes). Nothing changes, and the incident is escalated with the evidence.

"autonomy_change": { "quorum": 2, "chain": [{ "channel": "slack" }], "expires_s": 3600 }

Recent requestsCopy Anchor Link

Console → Approvals, or:

curl -s -H "Authorization: Bearer $TOKEN" $THREEAM/v1/approvals

Across a restartCopy Anchor Link

A waiting request is kept in approvals/<id>.json with the messages sent and the approvals given so far.

• Requests to raise autonomy or change the fix mode carry on at the next start (approval.resumed): nobody is asked again, the approvals already given count, approvals given while 3AM was down are read, and the request's clock keeps running from when it was first made.
• A fix waiting for approval is closed as abandoned, because its investigation stopped with the process. If the alert is still firing, a new incident opens.

Restore moves and drafted fixesCopy Anchor Link

A fix 3AM drafted itself, or a restore move when nothing is proven, is labelled in the request (Drafted fix …, Restore move (cause not proven)) with what it undoes, the inverse and the caveats, and needs two approvers by default (drafted_fixes.quorum) (Resolution).