Health, Doctor and preflight
Check that 3AM is up, that every part of it works, that its host is fit, and that it can reach every tool.
3AM Doctor
Doctor checks a running install part by part. Each check says what it found and, when something is wrong, what to do. Open 3AM Doctor in the console (also under the ? menu), or run it from the command line:
./install.sh --doctor # on a server, from the 3am-<version> folder
podman exec 3am-core /opt/3am/3am-core doctor # or: doctor --json, for automation
kubectl -n 3am exec statefulset/threeam -- /opt/3am/3am-core doctor| Area | Check | Healthy when |
|---|---|---|
| Core | 3AM | Operating (not in setup), and the runtime loaded without an error |
| Core | Alert intake | The intake loop ran in the last few poll intervals |
| Core | What-changed snapshots | 3AM read your systems' setup in the last 30 minutes |
| Core | Investigations | How many are running, and how many incidents are still firing |
| Licence | Licence, confirmed with Warden | Valid, not near expiry, and confirmed (or offline by choice) |
| Connections | Each connection | Its health check passes now, with how long it took |
| Models | Model placement | On a GPU or behind endpoints. Fails on a host without a GPU whose models are missing or local |
| Models | System 1, the investigator | Each answers (or is deliberately not configured) |
| Approvals | Waiting for a decision | Nothing pending for over an hour |
| Approvals | Escalation | Every step's channel is connected |
| Approvals | Lab auto-approver | Off. Fails while it is on: it approves everything |
| Policy | Fix mode, raising autonomy | Shown for information: advise or draft, and the quorum |
| Diagnosis | Accuracy | At least 90% of judged diagnoses were right |
| Audit | Ledger integrity | The hash chain and the seals verify |
| Storage | Data volume | At least 15% free |
The overall status is the worst check's: ok, warn (working, but needs attention soon) or fail
(something 3AM needs isn't working). The console's answer is cached for 15 seconds; Check again re-runs it.
GET /v1/doctor returns the same report (API). When you contact support, include Doctor's
report.
Is it up?
GET /health answers {"status": "ok"} without signing in. Use it from your load balancer or monitoring:
curl -s http://127.0.0.1:8700/healthThe container's own health check runs 3am-core health, which needs no curl.
GET /v1/status (also without signing in) says which mode 3AM is in: setup (no valid licence yet), configuring
(no connections yet) or operating, and why.
Preflight
Preflight checks the host and the network, before or after install:
3am-core preflight # host, licence, disk, accelerator, and a TCP connect to every configured tool
3am-core preflight --deep # also asks each connector for its health (credentials, permissions)
3am-core preflight --json # for automationEach line is ok, warn or fail, and the command exits non-zero on any fail. Run it after install, after every
upgrade, and whenever a tool changes its address or credentials.
Connectors
3am-core connectors list # every connector kind in this build, and its capabilities
3am-core connectors check # health of each configured connectorThe console's Connections page shows the same, with the last problem each connector reported.
Monitoring 3AM itself
| Watch | How |
|---|---|
| The service | GET /health every minute; alert after three failures |
| Everything else | GET /v1/doctor (with the admin token): alert when status is fail |
| The container | Podman and Docker's own health check (3am-core health, run directly, no shell): podman ps shows (healthy) |
| Disk | The data volume (/var/lib/3am): the audit log grows with incidents. Alert at 80% |
| The licence | GET /v1/status: licence.expires_at, and licence.in_grace once it has expired |
| The audit log | ledger-verify weekly (Logs and the audit log) |