Architecture and security

What runs where, what connects to what, where 3AM keeps its data, and how access, secrets and the audit log are protected.

3AM is one service inside your network. It connects out to your tools and your models, works out each incident's cause, and acts only on a proven cause, within your policy and with approval. Everything it does is written to a signed audit log before it has any effect.

YOUR NETWORKMonitoringPrometheus, Datadog…ApprovalsSlack, PagerDuty…Systems it fixesKubernetes, SQL, SSHCode and ticketsGitHub, ServiceNow…3AM server3am-coreconsole and APIinvestigationsconnectorsAudit logsignedData volume/var/lib/3amPeopleport 8700Modelsendpoint or GPUOpenRouteroptionalWardenwarden.3am.silicence check, hourlythe only call to 3AM
Arrows point the way each connection is opened. People opening the console, and optional alert webhooks, are the only connections into 3AM.

What runs

PartWhat it is
3am-core3AM itself: the console, the API, investigations and connectors, in one container
3am-modelOnly on a server with a GPU: runs the models locally, reachable only by 3am-core
3am-causalOptional (--causal): finds cause and effect between metrics that shifted together
Data volumeEverything 3AM keeps, mounted at /var/lib/3am

On Kubernetes the same images run from the Helm chart, with 3am-core as a single-replica StatefulSet, because the audit log has one writer.

Where data lives

Everything is in the data volume, so backing it up backs up 3AM (Backup and restore).

FolderWhat's in it
ledger/The audit log and its signing key
incidents/, episodes/Every incident, and each investigation in it
config/What you set in the console: connections, policy, estate corrections
secrets/Secrets entered in the console, one file each, readable only by 3AM
identity/People, teams, sessions and sign-in settings
snapshots/, learning/What 3AM has seen change, and what it has learned

Who can do what

  • People sign in with your identity provider (OIDC), your directory (LDAP), or a code sent by email. Their role (viewer, responder, admin or owner) is checked on every request (Roles).
  • Connectors use the least access that works. Kubernetes gets a role per namespace, with no exec and no Secrets. SSH runs only the commands a gate on the host allows. SQL checks run in read-only transactions.
  • Every change passes the same gates: System 1's check, a rehearsal, your policy, approval with the evidence, then verification. A halt switch stops everything at once.

Secrets

  • Typed into the console: saved to a file only 3AM can read, never shown again, never written to the audit log.
  • In configuration files: always a reference to a file or an environment variable. 3AM refuses a secret written inline.
  • Sent to a model: secrets, tokens and card numbers are masked first. Each model call is in the audit log, with the masked prompt.

The audit log

Each event carries the hash of the one before it, and the log is sealed and signed in segments with the install's own key. 3am-core ledger-verify finds any event that was edited, removed, inserted or reordered. Nothing in 3AM edits or deletes it. Logs and audit

The containers

Non-root, a read-only filesystem, every Linux capability dropped, no shell in the image. On RHEL, the UBI image uses the FIPS-validated OpenSSL.

What leaves your network

  • To your model endpoint: each investigation's evidence, masked. With your own endpoint, nothing leaves your network (What a hosted model sees).
  • To 3AM: the licence check, about once an hour, with signed usage counts. Licences are verified offline, so this can be turned off for air-gapped sites.

Nothing else. Your code, alerts and evidence stay with you.

On this page