PagerDuty

PagerDuty does three jobs for 3AM. Open incidents on the services you choose become 3AM incidents. Once 3AM may act on a service, it writes its diagnosis as a note on the page that woke someone up. And it can ask for approvals: 3AM opens an incident on an approval service, and your approvers answer with a note saying approve or reject.

Before you start

• A PagerDuty user with the Admin or Account Owner role, to create an API key.
• The IDs of the services 3AM should watch.
• Outbound HTTPS from 3AM to api.pagerduty.com (EU accounts: api.eu.pagerduty.com). PagerDuty never connects to 3AM.

1. Create a REST API keyCopy Anchor Link

Integrations → API Access Keys → Create New API Key, description 3AM. Leave Read-only API Key unticked: 3AM writes notes and approval incidents. Copy the key; PagerDuty shows it once.

You don't need an App or OAuth client. Add New App is for apps published to the PagerDuty marketplace.

2. Pick the user 3AM writes asCopy Anchor Link

Notes and approval incidents appear under a PagerDuty user. Create a service user such as 3am@bank.example (People → Users → Add Users) or reuse one. You'll enter its e-mail as from_email.

3. Find the service IDsCopy Anchor Link

A service's ID is in its URL: https://<you>.pagerduty.com/service-directory/PXXXXXX. Leave the list empty to watch every service.

4. Create an approval service (optional)Copy Anchor Link

Only if PagerDuty should carry approvals. Services → Service Directory → New Service:

• Name: 3AM approvals.
• Escalation policy: one that reaches the people who may approve 3AM's changes.
• Integrations: none. Only 3AM creates incidents here.

Note its ID for service_id.

5. Add the connectorCopy Anchor Link

In the console: Connections → Add a connection → PagerDuty. Or in connectors.json:

{ "name": "pagerduty", "kind": "pagerduty",
  "settings": { "api_token": { "file": "/etc/3am/secrets/pagerduty-token" },
                "from_email": "3am@bank.example",
                "watch_services": ["P7B0Q8Q", "P3KD92L"],
                "service_id": "P5ZUI27",
                "approvers": ["anna.k@bank.example", "raj.s@bank.example"] } }

6. VerifyCopy Anchor Link

Test connection authenticates, then looks up the user and each service:

Connected. authenticated; 3AM approvals (approvals); core-banking-db

Then send a test page to a watched service through its Events API v2 integration (the integration key is on the service's Integrations tab):

curl -s https://events.pagerduty.com/v2/enqueue -H 'Content-Type: application/json' -d '{
  "routing_key": "<integration key>", "event_action": "trigger", "dedup_key": "3am-test-1",
  "payload": {"summary": "MySQLReadOnly core-banking-db (3AM test)", "source": "db-1", "severity": "critical",
              "custom_details": {"alertname": "MySQLReadOnly", "service": "core-banking-db"}}}'

Within one poll interval (15 seconds by default) the console shows an incident for MySQLReadOnly. Resolve the test page by sending the same request with "event_action": "resolve".

How it behavesCopy Anchor Link

Pages in

• Triggered and acknowledged incidents on the watched services. 3AM's own approval incidents are never read as pages.
• One PagerDuty incident is one signal, even when alert grouping puts several alerts in it. The first alert's dedup key is kept as the dedup_key label.
• Severity comes from the priority (P1–P2 critical, P3–P4 warning, P5 info), otherwise from urgency.
• The alert name and labels come from the integration that raised the page (Alertmanager, Datadog, …).

Notes out

• Written only when 3AM may act on the service. In shadow mode nothing in PagerDuty changes.

Approvals

• Each request opens an incident with a dedup key, so a retry can't open a duplicate.
• High-risk requests ask for high urgency. Services with dynamic notifications decide urgency themselves.
• A note saying approve or reject from someone in approvers is a decision. PagerDuty notes carry the author's name, so 3AM looks up their e-mail.
• Two approvers on the same incident make a quorum of two.
• When the request closes, 3AM adds a final note and resolves the approval incident.

Rate limits

• On 429, 3AM waits as long as PagerDuty asks. On 5xx it backs off and retries. It never retries authentication or validation errors.

TroubleshootingCopy Anchor Link

the API key is wrong or revokedCopy Anchor Link

Create a new key (step 1) and paste it again.

the key lacks permissionCopy Anchor Link

The key is read-only. Create one with Read-only API Key unticked.

no PagerDuty user with e-mail …Copy Anchor Link

from_email doesn't match a user. Check People → Users.

service not foundCopy Anchor Link

A service ID is wrong. Copy it from the service's URL.

Pages don't appearCopy Anchor Link

The service isn't in watch_services, or the page was already open when 3AM started. Alerts already firing at start are treated as known, and they count again once they re-fire.

Approvals aren't countedCopy Anchor Link

The note's author isn't in approvers (compare e-mails), or the note doesn't contain approve or reject.

CertificationCopy Anchor Link

Live against a real PagerDuty account (2026-10-04): 16 live checks and 15 contract checks.

• A page raised through Events API v2 arrived with its alert name and labels.
• 3AM's note landed on the incident.
• An approval made by a second user was attributed to that user's e-mail, and the request was closed.