Connectors
What each connector gives 3AM, what "certified" means, and how to add one.
A connector is 3AM's link to one of your tools. Each declares what it can do, and 3AM uses it for exactly that:
| Capability | What 3AM does with it |
|---|---|
signal | Its alerts or failures open incidents |
telemetry | Queries it for evidence while proving a cause |
action | Makes approved changes through it |
approval | Asks people to approve or reject a fix |
source | Searches its repositories |
ticket, cmdb | Records incidents; reads service ownership |
List what your build includes with 3am-core connectors list:
alloy Grafana Alloy signal
datadog Datadog signal,telemetry
kubernetes Kubernetes signal,action
pagerduty PagerDuty signal,approval
prometheus Prometheus / Alertmanager signal,telemetry,action
slack Slack approval
victoriametrics VictoriaMetrics signal,telemetry,action
xmatters xMatters approval
…Certified
| Connector | Alerts in | Evidence | Fixes it can make | Approvals |
|---|---|---|---|---|
| PagerDuty | pages | notes on the page | yes | |
| Kubernetes | pod and rollout failures | yes | rollback, restart, scale, restart one pod, grow a volume; also carries out other packs' fixes | |
| Prometheus / Alertmanager | alerts | PromQL | silences; restarts and volume growth through Kubernetes | |
| VictoriaMetrics | alerts (vmalert, Alertmanager) | PromQL / MetricsQL | silences, reload; restarts and volume growth through Kubernetes | |
| Argo CD | degraded apps, failed or stuck syncs, Git errors, drift | deploy history: revision, author, commit | GitOps rollback, sync, stop a stuck sync; restarts its own components through Kubernetes | |
| Databases (MySQL, PostgreSQL) | down (and why), restarts, locks, connection leaks, read-only primary, replication | sessions, locks, replication, read-only queries | end a blocking session, close a leak, writable again, start replication; restart or grow through Kubernetes | |
| Servers (SSH) | an application down, host unreachable, disk full, host key changed | unit state, logs with the error line, disk, memory, ports, thread dumps | restart a service, clean old logs: only what each server's 3am-gate allows |
Certified means it passed four levels, on the real product:
- Contract. It handles the vendor's documented API: authentication, pagination, rate limits, errors.
- Live. The same checks against the real product, with the least permissions its guide asks for.
- Safety. Wrong credentials and unreachable hosts give clear messages. Secrets never reach the audit log, and rehearsals change nothing.
- End to end. Real failures reach the right cause and the right fix, or the right refusal to act, and the fix is verified.
Available, certification in progress
These work today and are being certified next. Their settings are on More connectors.
| Group | Connectors |
|---|---|
| Monitoring | Grafana, Grafana Alloy, Splunk, Datadog |
| Approvals | Slack, Symphony, xMatters, Twilio voice, e-mail |
| Systems | Oracle and SQL Server (reads and approved changes), HTTP automation |
| Code | GitHub, GitLab, Bitbucket, any Git server |
| Tickets and CMDB | Jira, ServiceNow (guide ready; live certification on Zurich next) |
Next to be certified: Grafana Alloy, xMatters and ServiceNow.
When a connector stops answering
3AM watches its own connectors: when one fails to answer, it records the failure, counts the consecutive failures, and
opens a connector.down incident naming the connector, its kind and the error it got.
Alert sources are checked every time 3AM reads alerts, because a source 3AM can't reach is a blind spot. Other
connections are checked too when you set k8s_deployment on them, so they can heal the same way.
- It recovers by itself. As soon as the connector answers again the synthetic signal stops firing and 3AM logs
signal.source_recovered, so the incident closes on evidence rather than on a timer. - It can fix it, if you let it. Set
k8s_namespaceandk8s_deploymenton the connector and 3AM offers a rolling restart of that Deployment through the Kubernetes connector. Low risk, reversible, and approved like any other action. - Otherwise it escalates. With no deployment configured, 3AM hands the incident to a person with the connector name, kind and error text.
- One broken connector never stops intake. The failure is per connector; the others keep polling.
Because the incident is a real one, it follows the normal path: the cause is proven, the fix is rehearsed, and nothing changes without approval.
Add a connector
In the console, Connections → Add a connection: fill in the form, select Test connection, then Save. Test connection uses the real API, so what it reports is what 3AM will see.
To manage connectors as code, list them in /etc/3am/connectors.json. Secrets are always references to a file or an
environment variable, never values. See Configuration files.