Evidence

ChecksCopy Anchor Link

A check is a query plus a comparison, something you could rerun by hand. Checks come in three kinds:

When a query has alternatives, 3AM uses the one your monitoring actually has data for. A metric that doesn't exist on your install never counts as "zero".

What changedCopy Anchor Link

Every connector that can describe itself is read every ten minutes, and again when an incident opens. 3AM keeps what changed and when, for seven days:

At an incident, the changes of the two hours before it are leads:

• Changes that cancel out don't count. A change that was undone before the incident nets to nothing.
• Constant change is ignored. A value that changes all the time is business as usual, not a change.
• Health isn't a cause. A target going down or a host becoming unreachable is a symptom: it's a lead, never a proven root cause.
• Upgrades reset the comparison. After 3AM is upgraded to a version that records more, the first reading is a new starting point, not a burst of changes.

How customer data is protectedCopy Anchor Link

Only small tables (at most 2,000 rows) with no date or time column are read. Tables that grow with the business, such as accounts, transactions and journals, almost always have dates, so they're left out. Whatever is read follows these rules:

• Never read: tables and columns named like secrets (tokens, passwords, sessions, keys, salts, hashes, one-time codes, PINs).
• Kept as-is: numbers, and codes such as REPAYMENT_LOAN. Also names in columns that hold a code, a type or the name of a configuration object (a role's name, a product's type).
• Kept as a short fingerprint: any other text, such as an address, a note or a description. A change is still detected, but the text itself never leaves the database.

So a removed permission reads as role_id=5 (Mobile_Channel) permission_id=219, and a customer's address never appears anywhere.

What is abnormalCopy Anchor Link

For the alerting service and everything it depends on, 3AM finds the metrics each one has and compares the last few minutes with the hour before:

• A shift counts when it's large compared with the metric's usual variation and a change of at least 20%. A flat metric creeping up by a few units doesn't count.
• Metrics with a status code (status="403") are compared per status, so failures replacing successes show up even when total traffic is flat.
• Each shift gets its start time. A cause has to begin before its symptom.

What the errors sayCopy Anchor Link

The database's own record of failing statements, and the error messages of your applications, are grouped into patterns, with numbers, addresses and quoted values set aside. Patterns 3AM hadn't seen before the incident began are marked new.

What counts as proofCopy Anchor Link

• Live data only. Runbooks, past incidents and model output can suggest where to look. Only checks against live systems confirm a cause.
• No data isn't proof. A check with no data, or no way to run on your install, is inconclusive and says why. It never counts as passed.
• It must be new. A cause whose checks also held 30 minutes before the alert, while all was well, is already true before the incident. It's reported, not acted on.
• Two independent lines outside packs. A change, or a model's idea, must also fit what the symptoms show.
• Look-alikes exclude each other. A crash after a release is a bad release only if the logs show no dependency errors.
• Partial answers are refused. If a backend answers with incomplete data (a cluster missing a node), the check is inconclusive.
• Values from alerts are checked. A namespace or workload name taken from an alert must be a plain name before any check uses it.

What's keptCopy Anchor Link

Each check's exact query, its raw result and its verdict go to the audit log under the incident's id:

{"type": "check.result", "trace_id": "ep-20261004T175412-592a91",
 "data": {"class": "db_read_only", "id": "read_only", "kind": "metric",
          "query": "prometheus: max(mysql_global_variables_read_only + mysql_global_variables_super_read_only)",
          "expect": {"op": ">", "value": 0}, "passed": true, "result": [{"labels": {}, "value": 2.0}], "elapsed_s": 0.03},
 "reason": "read_only > 0: met"}

Query one incident's checks with the API:

curl -s -H "Authorization: Bearer $TOKEN" \
  "$THREEAM/v1/ledger?type=check.result&trace=ep-20261004T175412-592a91&order=asc"

Changes are in the audit log too (snapshot.changed), and so is every cause that was already true before an incident (check.chronic).