Written by: Nimesh Chakravarthi, Co-founder & CTO, Struct
Key takeaways from the webhook-to-investigator pattern
- The webhook-to-investigator pattern routes Alertmanager alerts through data collection, AI analysis, and structured diagnosis. This flow reduces first-pass triage time by 80%.
- Every alert rule must include a
runbook_urlannotation so the AI layer grounds its reasoning in documented procedures before emitting a diagnosis. - Alertmanager routes to a webhook receiver with a priority queue that returns 202 Accepted immediately, which prevents retries while the AI processes the alert.
- Read-only AI guardrails ensure the system only suggests remediation steps. Humans retain execution authority and the agent never writes to production systems.
- Struct automates your on-call runbook by embedding incident resolution verification that confirms fixes against live observability data before closing the loop.
Webhook-to-investigator pattern for Prometheus alerts
When Alertmanager fires, it sends a JSON payload to a configured webhook receiver. That receiver becomes the entry point for automated investigation. It enqueues the alert, runs read-only queries against Prometheus, Loki, and the Kubernetes API, feeds the collected context plus the alert’s runbook_url annotation into an AI reasoning layer, and emits a structured diagnosis. This pattern removes humans from the data-collection loop.
incident.io’s SRE tools guide reports that the gap between an alert firing and troubleshooting starting often runs 10–15 minutes for many teams. Coordination overhead such as manually creating Slack channels, identifying service owners, and opening runbooks consumes that time. A CNCF case study from April 2026 found a two-person SRE team spending 15–20 minutes of manual correlation per Prometheus alert before wiring an event-driven AI agent to handle first-pass investigation. The webhook-to-investigator pattern removes that dead time.
The Catchpoint SRE Report highlights the growing operational burden on SRE teams and how it limits time for technical training. Automating the first-pass investigation directly gives that capacity back.
Build an automated Prometheus alert investigator in under 10 minutes
The architecture below shows the complete pipeline. Each numbered step maps to one component in the flow.
Prometheus → Alertmanager → Webhook Receiver (202 Accepted) → Priority Queue → Data Collection (PromQL + Loki + K8s API) → Runbook Fetch (runbook_url annotation) → Read-Only AI Layer → Structured Diagnosis → Incident Resolution Verification Loop → Slack / PagerDuty Handoff- Annotate alert rules with runbook URLs.
Every Prometheus alert rule must carry a
runbook_urlannotation before automation adds real value. An alert without a runbook is an invitation to panic. The recommended annotation contract includessummary,descriptionwith the interpolated firing value,runbook_url, anddashboard_url:groups: - name: api rules: - alert: ApiHighErrorRate expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05 for: 5m labels: severity: critical team: backend annotations: summary: "High error rate on {{ $labels.service }}" description: "Error rate is {{ $value | humanizePercentage }} on {{ $labels.service }}" runbook_url: "https://runbooks.example.com/api-error-rate" dashboard_url: "https://grafana.example.com/d/abc/api-overview?var-service={{ $labels.service }}"A recommended CI linter combines
promtoolwith a script that fails a pull request when any alert is missing a required annotation or references a runbook file that does not exist. - Configure Alertmanager to route to the webhook receiver.
Add a webhook receiver alongside your existing PagerDuty or Slack routes. Use
continue: trueso humans are still paged while the investigator runs in parallel. Group byalertnameandnamespace, and setmax_alerts: 5on the AI agent receiver to prevent alert storms from flooding the investigation queue.route: group_by: ['alertname', 'namespace'] receiver: pagerduty routes: - match: severity: critical receiver: ai-investigator continue: true receivers: - name: ai-investigator webhook_configs: - url: 'http://investigator.internal:8080/webhook' max_alerts: 5 send_resolved: true - Build the webhook receiver with a priority queue.
The receiver must return
202 Acceptedimmediately so Alertmanager does not retry during LLM processing latency. A boundedasyncio.PriorityQueuewith a default max of 1,000 and priority derived from the severity label, such ascritical=0,warning=2,info=3, prevents accumulation behind slow enrichment calls. A minimal Python handler:@app.post("/webhook", status_code=202) async def receive_alert(payload: AlertmanagerPayload, background_tasks: BackgroundTasks): for alert in payload.alerts: priority = {"critical": 0, "warning": 2, "info": 3}.get( alert.labels.get("severity", "info"), 3 ) await queue.put((priority, next(counter), alert)) return {"status": "queued"}A 50-line circuit breaker that opens after five consecutive LLM failures and short-circuits calls for a 30-second cooldown keeps the on-call experience graceful. The system falls back to the unenriched baseline instead of failing silently.
- Collect data from Prometheus, Loki, and Kubernetes.
For each dequeued alert, run read-only queries scoped to the alert’s labels. Example PromQL queries for an
ApiHighErrorRatealert onservice=checkout:# Error rate over the last 30 minutes rate(http_requests_total{service="checkout",status=~"5.."}[30m]) # P95 latency histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{service="checkout"}[5m])) # Pod restart count kube_pod_container_status_restarts_total{namespace="production",pod=~"checkout-.*"}Companion Loki query to pull correlated log lines:
{namespace="production", app="checkout"} |= "error" | json | line_format "{{.timestamp}} {{.level}} {{.msg}}" | last 200Fetch the
runbook_urlannotation from the alert payload and retrieve the runbook content. The AI layer then grounds its reasoning in your team’s documented procedures. - Apply read-only AI guardrails.
AI agents should default to read-only for observability systems. State-changing remediation actions such as scaling services or rolling back deployments require specific elevated roles with extra approval workflows. Implement a suggest-only mode so the agent proposes remediation steps and a human with the appropriate role executes them.
Input validation for agents that convert alert text into PromQL or Loki queries must normalize and bound user input to prevent malformed queries. Before executing any query, verify the request is allowed for the agent role to prevent privilege escalation. Then scan for prompt injection signals that could manipulate the AI’s reasoning. Finally, enforce a strict schema for tool arguments by rejecting unknown fields, enforcing enums and ranges, and blocking local network IP ranges to prevent SSRF.
Log every blocked action with a reason code that includes policy ID, validator, tool, and triggering field. This supports auditability and continuous improvement of guardrails.
- Emit a structured diagnosis.
The AI layer outputs a JSON object that the Slack or PagerDuty handoff template consumes directly:
{ "alert": "ApiHighErrorRate", "service": "checkout", "severity": "critical", "firing_value": "8.3%", "root_cause_hypothesis": "Upstream payment-gateway returning 503 since deploy abc123 at 14:32 UTC", "supporting_queries": [ "rate(http_requests_total{service='checkout',status=~'5..'}[30m])", "{namespace='production', app='checkout'} |= 'error'" ], "runbook_url": "https://runbooks.example.com/api-error-rate", "suggested_action": "Roll back deploy abc123 or check payment-gateway health endpoint", "enrichment": "grounded", "resolution_verified": false } - Hand off to Slack and PagerDuty.
Pass the
runbook_urlthrough Alertmanagerdetails, extract it via PagerDuty Event Orchestration, and map it to a custom incident field so the on-call engineer can jump straight to instructions. In Slack, render the diagnosis as a threaded reply in the alert channel so engineers see root cause, supporting evidence, and suggested action without leaving their communication hub.
The seven-step pipeline above handles alert ingestion through diagnosis handoff. It still leaves one critical question: how your team confirms that the chosen fix actually resolved the incident in live systems.
Incident resolution verification with live observability data
Automated investigation without closed-loop verification leaves a critical gap. The system fires, the engineer acts, and no mechanism confirms the fix actually worked against live observability data. Incident resolution verification closes that loop by re-running the same PromQL and Loki queries used during investigation on a roughly one-minute polling interval. The incident is marked resolved only when the metrics return to baseline.
Struct’s Incident Tracker, launched August 3, 2026, is the flagship implementation of this pattern. It maintains active incident state, keeps status current automatically, and runs a roughly one-minute automated verification loop against observability data to confirm an incident is actually resolved, not just acknowledged. This capability differs from alert routing or AI enrichment. It performs continuous re-evaluation of real signal instead of a one-shot diagnosis.
The operational impact is measurable. Arcana, a Series A fintech with over 40 engineers, reduced average investigation time from 30 minutes to 2 minutes, reclaimed 56 developer hours per month, and runs 2,100+ automated investigations monthly after integrating Struct with Sentry, GitHub, GCP Cloud Logging, and Slack. Large-scale customers report an 80% reduction in triage time across thousands of monthly investigations.
The verification loop also surfaces a second category of value: alert quality improvement. When the system observes that a metric recovered without human intervention, it flags the alert as a candidate for threshold tuning or suppression. That signal flows back into the alerting layer at the pull request and deploy level through Struct’s Deploy Guard capability.
Frequently asked questions about Struct and this pattern
What is the minimum tooling maturity required to implement this pattern?
Your team needs Prometheus and Alertmanager already running, at least one log aggregation source such as Loki, CloudWatch, GCP Logging, or Datadog, and a Slack workspace or PagerDuty account for handoff. Struct connects to these via authenticated integrations and does not require you to instrument your application code or change your existing alert rules beyond adding runbook_url annotations. If your system lacks basic logging, trace IDs, or alerting triggers, automated investigation cannot deduce system state from code analysis alone. The golden starting point is a team already using Sentry or cloud logs alongside Prometheus.
How long does rollout actually take?
Connecting integrations and running the first automated investigation takes under 10 minutes. You authenticate your alert source such as Slack or PagerDuty, your code repository such as GitHub, and your observability context such as Prometheus, Loki, Datadog, or cloud logs, then enable auto-investigations. Struct’s white-glove onboarding is included in the 30-day risk-free pilot, so your team does not configure the pipeline alone.
What if our telemetry is sparse or our logging is inconsistent?
Struct’s investigation quality scales with the telemetry available. Teams with sparse logging receive narrower diagnoses. The AI layer reports what the data supports and does not hallucinate missing context. A practical approach starts with your highest-frequency alerts. Add runbook_url and description annotations with interpolated values to those rules first, then expand coverage as logging matures. Struct’s composable runbook encoding lets you specify exactly which correlation IDs and log patterns to look for. This partially compensates for inconsistent telemetry by directing queries precisely.
Does this architecture satisfy SOC 2 and HIPAA requirements?
Struct is SOC 2 Type II and HIPAA compliant, as documented at trust.struct.ai. Logs are accessed and processed ephemerally, and Struct does not store them beyond the investigation window. The read-only AI guardrail design means the agent never writes to production systems. If your organization requires full on-premise deployment with zero data leaving your VPC, Struct’s Enterprise tier includes sidecar and on-prem support options. Contact the team to assess fit before starting a pilot.
Can we encode our existing on-call runbooks into the investigator?
Struct accepts custom instructions, correlation ID formats, and copy-pasted internal runbook content directly. The AI follows your documented operational procedures when an alert fires, including team-specific escalation paths, service dependency maps, and known failure modes. Composable widgets let you guarantee that specific visual data, such as a particular Grafana panel or a specific Loki query, is always pulled for certain alert types. This behavior replicates what your most experienced engineers would check first.
Conclusion: Struct as your investigation and verification layer
The webhook-to-investigator pattern, built from an Alertmanager webhook receiver, a priority queue, read-only PromQL and Loki data collection, runbook-grounded AI reasoning, structured diagnosis output, and closed-loop incident resolution verification, reduces first-pass triage time and removes manual coordination overhead documented earlier in the article.
Arcana’s results, detailed in the verification section above, demonstrate the measurable impact of closed-loop automation on top of existing observability tooling.
Struct sits as an investigation and verification layer above Prometheus, Loki, Datadog, Sentry, and your cloud logs. It does not replace your observability stack. It automates the first pass, verifies resolution against real signal, and hands off clean context to the engineer who needs to act.
