Troubleshooting
Common problems with a running 3AM, what causes them, and how to fix them. Start with Doctor.
Start with 3AM Doctor. Open it in the console (the ? menu), or run ./install.sh --doctor on the server. It
checks every part of 3AM and says what to do about anything wrong (Health).
To check whether 3AM can reach each of your tools, run preflight:
podman exec 3am-core /opt/3am/3am-core preflight --deepok data volume writable /var/lib/3am
ok clock UTC now 2026-10-04 19:05
ok model placement no GPU: models behind endpoints
ok reach prometheus (prometheus.example.internal:9090) reachableAny WARN or FAIL line names what to fix, and the command exits non-zero on a FAIL.
When you contact support, include the output of preflight --deep --json and 3am-core version.
Signing in
On a server, copy it out of the data volume:
podman cp 3am-core:/var/lib/3am/admin-token - | tar -xO; echoOn Kubernetes:
kubectl -n 3am get secret threeam-admin -o jsonpath='{.data.token}' | base64 -d; echoEmailed codes need a mail provider: Settings → E-mail sign-in, with Send me a test e-mail to check it. With Resend, the sender must be on a domain you've verified in Resend. Until mail works, sign in with the admin token.
There's no valid licence. Settings → Licence, or curl -s http://127.0.0.1:8700/v1/status, says why. Each reason
is explained in Licence troubleshooting.
Alerts and incidents
Work down this list:
- Give it a moment. Your alerting adds its own delay first: the rule's
for:time and Alertmanager's grouping. - It may have joined an open incident. An alert that the open incident's cause explains joins it.
- Is its source healthy? Check Connections.
- Is the service turned off? Check Autonomy.
- Was it already firing when 3AM started? Those are treated as known until they fire again.
- Did it reach 3AM at all? Search the audit log:
curl -s -H "Authorization: Bearer $TOKEN" "http://127.0.0.1:8700/v1/ledger?type=signal.&q=MySQLReadOnly&limit=5"The incident's decision says why (Incidents):
shadow: the service is in shadow mode, so 3AM records the fix and asks no one.ask_human: the fix needs a judgement call, such as raising a memory limit, so 3AM sends advice instead.vetoed: System 1 found the evidence doesn't support the fix.denied,no_approval_route,dry_run_failed: your policy refused it, nobody could approve it, or the rehearsal failed.
It has a lead it couldn't prove, and it never acts on a lead. The incident lists what it tried. Usually the check that would prove it needs a connection you haven't added, and the cause says which.
Its checks also passed half an hour before the alert, while things were healthy, so it isn't what broke now. It's a standing problem to fix when convenient. 3AM keeps looking for this incident's cause.
It had no data: a metric your exporter doesn't publish, or a connection that isn't set up. The check says which. Missing data never counts as proof.
The model didn't answer in time, or its note was rejected because System 1 found a sentence the evidence doesn't support. The incident page shows which. Nothing else is affected. Check the models with Doctor.
Connections
The message says what to check:
- nothing is answering or didn't answer in time: a firewall, or a wrong address. Test from the 3AM server itself.
- refused the credentials: the password or token is wrong or expired.
- refused access: the account needs more permission. The connector's page lists exactly what it needs.
Not answering means no sync for 3 minutes: check its server. Gone? means a day of silence: if it was rebuilt or destroyed, remove it on the home and add the new site (Keychain).
Installing and upgrading
The models never run on the CPU. Choose your own endpoint or OpenRouter when asked (Models).
The installer printed the new 3AM did not start, and started the previous version again with its data and settings. Nothing is lost. Read the log it printed, fix what it says (often a key file that moved), and run the installer again.
Treat it as a security incident. Leave the data volume as it is, restore the last good backup on a separate server to compare, and contact support@3am.si.