Prometheus and Alertmanager

Firing alerts from Alertmanager become 3AM incidents (silenced and inhibited ones don't). 3AM's packs run their checks as PromQL against your Prometheus: MySQL, Kubernetes, JVM, nginx and the rest. 3AM also watches Prometheus and Alertmanager themselves. When monitoring is broken, its own evidence is unreliable, so it treats that as an incident too.

Before you start

• Read access to Prometheus' HTTP API, and to Alertmanager's.
• Write access to Alertmanager silences, only if 3AM may create silences. Every silence is approved.
• Outbound access from 3AM to both. Nothing connects to 3AM.

1. Create credentialsCopy Anchor Link

web.yml stores bcrypt hashes. Make one for 3AM's password:

htpasswd -nbBC 10 3am 'a-long-random-password' | cut -d: -f2

basic_auth_users:
  3am: $2y$10$…          # the hash from above

2. Check your exportersCopy Anchor Link

The packs' checks read standard exporter metrics. A stock exporter covers the core checks; a few need a flag:

A check whose metric is missing is reported as inconclusive. It never counts as passed.

3. Add the connectorCopy Anchor Link

In the console: Connections → Add a connection → Prometheus / Alertmanager. Or in connectors.json:

{ "name": "prometheus", "kind": "prometheus",
  "settings": { "url": "https://prometheus.bank.internal:9090",
                "username": "3am", "password": { "file": "/etc/3am/secrets/prometheus-password" },
                "alertmanager_url": "https://alertmanager.bank.internal:9093",
                "alertmanager_bearer_token": { "env": "AM_TOKEN" },
                "ca_file": "/etc/3am/secrets/bank-ca.pem" } }

4. VerifyCopy Anchor Link

Test connection runs a real query (not just a ping) and checks Alertmanager with its own credentials:

Connected. Prometheus 3.15.0 + Alertmanager 0.34.1
jobs: 6 (mysql, prometheus, …)   alert rules: 95   targets: 32

To check the whole path, add a rule that always fires and reload Prometheus:

groups:
  - name: 3am-test
    rules:
      - alert: ThreeAMPipelineTest
        expr: vector(1)
        labels: { severity: critical, service: monitoring }

Within a minute the console shows an incident for ThreeAMPipelineTest. 3AM finds nothing to prove and says so. Remove the rule afterwards.

What it proves and fixesCopy Anchor Link

3AM raises these from Prometheus' and Alertmanager's own status, even with no alert rule for them. If your own rule for the same thing fires, 3AM uses yours.

Restarts and volume growth go through the Kubernetes connector, which finds the workload behind Prometheus', Alertmanager's or the target's address. Without it these causes go to a person with the evidence.

ActionsCopy Anchor Link

TroubleshootingCopy Anchor Link

Prometheus: authentication failedCopy Anchor Link

Check the user and password, or the bearer token.

Alertmanager: authentication failedCopy Anchor Link

Alertmanager has its own credentials. Fill in the alertmanager_* settings.

the certificate is not trusted: set ca_fileCopy Anchor Link

Set CA certificate to your internal CA, in PEM format.

Prometheus refused the query (bad_data)Copy Anchor Link

A check's PromQL didn't parse on your Prometheus. The message includes Prometheus' own explanation; send it to support@3am.si.

Silenced alerts still appearCopy Anchor Link

Without alertmanager_url, 3AM reads Prometheus' alerts, which know nothing about silences. Add the Alertmanager URL.

CertificationCopy Anchor Link

Live, on Prometheus 3.15 behind HTTPS with an internal CA and basic auth, Alertmanager 0.34 behind a bearer-token proxy, and the real mysqld_exporter 0.20 (2026-10-04):

• 46 checks. End to end, the database was set read-only. This Prometheus fired MySQLReadOnly through Alertmanager, 3AM proved it from this Prometheus' metric and fixed it with approval, and the alert resolved.
• Fixes, on Kubernetes. A hung Prometheus, a hung Alertmanager and a hung scrape target were each restarted and verified (details).