Kubernetes

With this connector 3AM watches the namespaces you choose, works out why a workload is failing from the API server alone (no metrics stack needed), and fixes it with your approval. It is also how fixes found by other connectors get carried out: when VictoriaMetrics or Prometheus proves a component is hung, this connector restarts it.

Before you start

• kubectl access as a cluster admin, once, to create a ServiceAccount and a Role.
• The namespaces 3AM should watch and act in.
• Network access from 3AM to the API server (inside the cluster that's https://kubernetes.default.svc).

1. Create a ServiceAccount and a RoleCopy Anchor Link

3AM's access is one ServiceAccount, plus a Role in each namespace it watches. Both manifests are in the kubernetes/ folder of each 3AM release. Installed from the Helm repository? curl -fsSL https://get.3am.si | sh downloads a release without installing it.

kubectl apply -f kubernetes/3am-rbac.yaml           # namespace 3am, the ServiceAccount, a token, StorageClass read

for ns in bank-prod payments-prod; do
  sed "s/WATCHED_NAMESPACE/$ns/" kubernetes/3am-role.yaml | kubectl apply -f -
done

rules:
  - apiGroups: [""]
    resources: [pods, pods/log, services]
    verbs: [get, list, watch]                 # see failures; find which workload an address belongs to
  - apiGroups: [apps]
    resources: [deployments, statefulsets, daemonsets, replicasets]
    verbs: [get, list, watch]                 # map pods to workloads, read revisions
  - apiGroups: [apps]
    resources: [deployments, statefulsets, daemonsets, deployments/scale, statefulsets/scale]
    verbs: [patch]                            # restart, scale, roll back
  - apiGroups: [""]
    resources: [pods]
    verbs: [delete]                           # restart_pod: the controller recreates the pod
  - apiGroups: [""]
    resources: [persistentvolumeclaims]
    verbs: [get, list, patch]                 # expand_volume
  - apiGroups: [""]
    resources: [events]
    verbs: [create]                           # 3AM's diagnosis as an Event on the workload (optional)

2. Collect the token, CA and API addressCopy Anchor Link

kubectl -n 3am get secret 3am-connector-token -o jsonpath='{.data.token}' | base64 -d > 3am-k8s-token
kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > k8s-ca.pem
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'; echo

On EKS, AKS and GKE, use the address and CA from your kubeconfig as they are. 3AM authenticates with the ServiceAccount token, so a cloud auth plugin in your kubeconfig doesn't matter.

3. Add the connectorCopy Anchor Link

In the console: Connections → Add a connection → Kubernetes. Or in connectors.json:

{ "name": "k8s", "kind": "kubernetes",
  "settings": { "api_url": "https://10.0.0.1:6443",
                "token": { "file": "/etc/3am/secrets/3am-k8s-token" },
                "ca_file": "/etc/3am/secrets/k8s-ca.pem",
                "namespaces": ["bank-prod", "payments-prod"] } }

4. VerifyCopy Anchor Link

Test connection asks the API server, namespace by namespace, whether the token can do each thing 3AM needs:

Kubernetes v1.34.0; permissions present in bank-prod, payments-prod

A missing permission is named, with the namespace and what it's for:

Kubernetes v1.34.0, but the token cannot: list pods in payments-prod (see pod state)

What it detectsCopy Anchor Link

The names match kube-prometheus, so if your Prometheus already alerts on these, the same causes apply. Three crash-looping replicas of one workload are one incident.

What it proves and fixesCopy Anchor Link

"Bad release" and "dependency down" exclude each other: restarting pods whose database is down is the classic wrong move. How fixes are approved and verified is the same for every connector: see How 3AM decides.

ActionsCopy Anchor Link

Every change runs first with dryRun=All: the API server and your admission webhooks validate it and nothing changes. 3AM refuses namespaces outside its list, and the Role would refuse them anyway.

TroubleshootingCopy Anchor Link

the token is invalid or expiredCopy Anchor Link

Re-read the token from the 3am-connector-token Secret (step 2). Tokens from kubectl create token expire.

the API server's certificate is not trusted: set ca_fileCopy Anchor Link

Set CA certificate to k8s-ca.pem from step 2.

the token cannot: list pods in <namespace>Copy Anchor Link

The Role isn't applied in that namespace. Run the loop from step 1 for it.

An action fails with the 3AM Role does not allow thisCopy Anchor Link

That namespace's Role is missing a rule, or you removed it on purpose. Re-apply 3am-role.yaml if you didn't.

A failing pod opens no incidentCopy Anchor Link

Its namespace isn't in Namespaces.

CertificationCopy Anchor Link

Live, on Kubernetes 1.34 with exactly this RBAC and only 3AM's token (2026-10-04):

• Connector, 32 checks. Signals, workload resolution and health reporting.
• End to end, 30 checks. Two broken releases (a config panic, a missing image) were rolled back and verified healthy. A release whose database was down was left alone. An OOM and a missing ConfigMap got advice, and nothing ran.
• Fixes for other connectors, 54 checks. With 3AM inside the cluster, hung components were each located, restarted and verified: vmagent, one vmstorage pod, Alertmanager, Prometheus and two exporters. A volume-growth request on a driver that can't grow volumes was reported as failed, not as done.