Written by: Nimesh Chakravarthi, Co-founder & CTO, Struct
Key Takeaways
- Agentic on-call systems autonomously detect, investigate, and remediate incidents using a continuous reasoning loop, reducing MTTR by compressing manual investigation to under five minutes.
- Autonomy levels (L0–L4) define how much an agent can act independently versus requiring human approval, with most production teams operating at L2 or L3.
- Incident resolution verification automatically confirms fixes by querying live observability data after remediation, which prevents premature incident closure and silent re-opens.
- Core architecture includes perception, cognition, action, and verification layers, supported by shared knowledge graphs, multi-agent coordination, and strict blast-radius controls.
- Teams using Struct report up to 80% faster triage and 17x investigation coverage; start a 30-day Struct pilot to see these gains on your own incidents.
Why agentic on-call systems matter for MTTR
Agentic on-call systems matter for MTTR because they compress the investigation phase, historically the longest part of incident resolution, from 30–45 minutes of manual log-hunting to under five minutes of automated correlation. Many incident responders receive high volumes of alerts per shift, with only a small percentage genuinely warranting attention. This pattern drives a well-documented burnout cycle: on-call stress has impacted burnout and attrition for many SREs, with substantial time spent on operations.
The financial stakes are concrete. The median enterprise IT outage costs $9,000 per minute according to ITIC’s 2025 survey, so a 45-minute investigation before any fix begins can cost over $400,000. Teams using AI-driven operations or automated incident workflows commonly reduce MTTR by 30–70%. Practitioners on Reddit SRE threads consistently describe the same pattern: senior engineers become the single point of failure for triage because newer hires lack the tribal knowledge to navigate five disparate tools at 3 a.m.
Agentic systems break that dependency by encoding runbook logic into agents that execute the first-pass investigation automatically. Every engineer, regardless of tenure, receives a structured starting point within minutes of an alert firing.
How autonomy levels map to real production teams
Autonomy levels map to production teams by defining which actions an AI agent executes independently versus which require a human approval gate. Frameworks define a five-level ladder from fully manual to full autonomy.
The table below maps the five levels to their operational characteristics and typical team fit:
| Level | Label | Agent Behavior | Human Role | Typical Team Fit |
|---|---|---|---|---|
| L0 | Manual | No automation; humans do everything | Full ownership of all actions | Pre-AIOps teams |
| L1 | Assisted | Automates monitoring and investigation; surfaces findings | Approves and executes every action | Teams new to AI-assisted triage |
| L2 | Partial Autonomy | Investigates and proposes a mitigation plan | Reviews plan; approves before agent actuates | Series A–B teams building confidence |
| L3 | High Autonomy | Automates investigation, approval, and actuation for well-defined scenarios | Reviews outcomes; sets policies | Most Series B–C SaaS and fintech teams |
| L4 | Full Autonomy | Plans, executes, and adapts multi-step resolutions across the full incident lifecycle | Handles escalations and novel failures only | Mature platform teams with proven runbooks |
The golden rule for advancing autonomy is that an action is only promoted to autonomous (L4) after it has proven safe many times at the manual-approval (L3) level. Certain low-risk remediation classes such as rollbacks inside rollout windows, isolated restarts, and scale-outs can qualify for autonomous execution. Database migrations and multi-service rollbacks typically require human approval.
See how Struct maps to your team’s autonomy level in a focused 30-minute demo.
Incident resolution verification
Autonomy levels define how much an agent can act independently, but they do not explain how the agent knows an action succeeded. Incident resolution verification is the closed-loop process by which an agentic system automatically confirms that an incident is actually resolved by querying live observability data after a remediation action, rather than relying on a human to manually check dashboards. Without this loop, teams close incidents prematurely. A restarted pod may recover for 90 seconds before the underlying memory leak resurfaces, and no alert fires until the next customer-facing degradation.
A dedicated Verification Agent provides closed-loop assurance by executing post-action health checks, comparing device state against expected outcomes, monitoring for regression during a configurable bake-in period, and triggering automatic rollback on verification failure. This behavior is architecturally distinct from simply logging that a remediation command was issued.
Struct's Incident Tracker implements incident resolution verification as a roughly one-minute automated loop. After a remediation step is applied, Struct continuously queries connected observability sources such as Datadog, Grafana, Sentry, and cloud logs, then updates incident status only when the signal data confirms the condition is cleared. If metrics regress during the bake-in window, the incident remains open and the on-call engineer is notified. This approach prevents the silent re-open problem that plagues teams relying on manual confirmation.
Core architectural layers of an agentic on-call system
An agentic on-call system is organized into four functional layers that separate data ingestion, reasoning, execution, and governance. Each layer has a distinct responsibility boundary, and crossing those boundaries requires explicit contracts.
The text-based architecture below shows how each layer maintains strict responsibility boundaries while enabling end-to-end incident resolution, and how the verification layer closes the loop by feeding results back to observability rather than assuming success:
[Alert Sources: PagerDuty / Slack / Sentry] │ ▼ [Perception Layer] Event streaming backbone Anomaly detection agents Alert correlation and deduplication │ ▼ [Cognition Layer] Shared knowledge graph (services, SLOs, prior incidents) Detector Agent → Hypothesis Agent → Validator Agent Planner Agent → Critic / Governance Agent │ ▼ [Action Layer] Remediation execution via cloud and Kubernetes APIs Blast-radius limits · Canary rollouts · Auto-rollback Human approval gate for high-risk operations │ ▼ [Verification Layer] Post-action health checks against observability data Bake-in period monitoring Rollback trigger on regression · Incident record update │ ▼ [Observability and Governance Plane] Step-level traces · Tamper-evident audit log Policy engine · Kill switch · Cost controls
The shared state and context layer functions as persistent agent memory. It provides a continuously updated view of aggregated system state that all agents access via streams or materialized views rather than private memory. 41% of surveyed software organizations report limited or broad production use of MCP servers.
The governance plane spans all layers. Evidence in 2026 agent systems includes step-level traces, a tamper-evident log of intent, approval, action, and result, an eval harness wired into CI/CD, and cost controls with a kill switch. The model itself is the most replaceable component. The runtime, tools, policies, and evidence layer are owned long-term.
How multi-agent teams coordinate during an incident
Multi-agent teams coordinate by passing structured messages through a shared event backbone rather than making direct API calls between agents. This pattern preserves temporal decoupling, fault isolation, and full auditability. The Cognition Layer uses specialized roles: a Detector Agent groups alerts into incidents, a Hypothesis Agent proposes root causes, a Validator Agent issues targeted queries to observability tools to confirm or reject hypotheses, a Planner Agent creates remediation plans with preconditions and rollback triggers, and a Critic Agent reviews plans against organizational safety policies.
The Meta-Agent framework (arXiv:2605.25233) decomposes tasks into a directed acyclic graph of agent specifications with explicit input/output contracts and verification criteria, then uses a coordinator to dispatch subtasks while execution-time verification gates intermediate outputs. The VMAO framework (arXiv:2603.11445v2) extends this with a Plan-Execute-Verify-Replan loop, using a ResultVerifier that produces a completeness score, missing aspects, contradictions, and a recommendation of accept, retry, or escalate for each intermediate result.
For multi-agent coordination, Google's A2A protocol under Linux Foundation governance handles agent-to-agent work alongside MCP, forming a layered stack where MCP gives agents tool access and A2A enables team coordination. Struct's composable architecture encodes team-specific runbooks into this coordination layer, so the Hypothesis and Validator agents follow the same investigation sequence a senior engineer would.
Safety boundaries and blast-radius controls for production use
Safety boundaries for agentic on-call systems rely on four concentric controls: tool scope whitelisting, data scope enforcement, reversibility classification, and human approval gates for irreversible actions. Concentric containment rings bound autonomous agent actions through API endpoint whitelisting, row-level security and read-replicas, compensating transactions and rollback, and cryptographic human approval for irreversible changes, as outlined in the Five Eyes 2026 guidance on agentic AI.
Circuit breakers deserve special attention. Circuit breakers must operate outside the agent control plane because an agent can suppress its own safety reporting if controls live inside the same plane. Galileo AI research found a single compromised agent can affect 87% of downstream decision-making within four hours, faster than most incident response processes can initiate.
The practical implementation for a 15–80 engineer team running Struct applies the concentric containment model as a graduated autonomy ladder. Read and propose actions execute automatically with full audit logging, which establishes the baseline observability layer. Reversible remediations such as pod restarts, cache clears, and feature-flag rollbacks execute at L3 with post-action verification because teams can roll them back if verification fails. Irreversible actions such as database migrations, IAM changes, and multi-service rollbacks route to a Slack approval gate before execution, which preserves the human-in-the-loop for changes that cannot be undone. A kill switch then halts all agent execution independently of code deployment, providing an out-of-band safety mechanism that operates outside the agent control plane.
How Struct, PagerDuty, Datadog, Resolve AI, and Harness compare
With the architectural and safety requirements established, the next step is choosing a platform that delivers these capabilities without a heavy enterprise rollout. The table below answers the decision most teams face: which platform can provide automated investigation quickly, across existing tools, while still verifying outcomes. The comparison focuses on setup friction, cross-tool investigation scope, and verification capabilities, which together determine time-to-value for lean engineering teams.
| Capability | Struct | PagerDuty AIOps | Datadog Bits AI | Resolve AI / Harness |
|---|---|---|---|---|
| Setup time | Under 10 minutes; no enterprise deployment required | Days to weeks; requires PagerDuty ecosystem configuration | Requires existing Datadog instrumentation; hours to days | Enterprise sales process; lengthy indexing and onboarding |
| Investigation scope | Cross-stack: Datadog, Sentry, cloud logs, GitHub, traces correlated in one investigation | Alert correlation and noise reduction; limited cross-tool RCA | Datadog telemetry only; does not ingest external sources natively | Broad integrations but requires significant configuration per environment |
| Incident resolution verification | ~1-minute automated verification loop against live observability data; auto-rollback on regression | Manual confirmation; no automated closed-loop verification | No dedicated verification loop; relies on monitor state changes | Varies by deployment; not a named product capability |
| Slack-native workflow | Fully Slack-native; streams updates in real time; supports follow-up questions in-thread | Slack integration available; primary interface is PagerDuty web app | Slack notifications; investigation UI lives in Datadog | Slack integration; primary interface is vendor platform |
Struct is positioned as an investigation layer that sits on top of existing observability tooling rather than replacing it. Struct auto-dedupes related alerts with no configuration and runs one investigation per real incident, which directly addresses the alert fatigue problem without requiring teams to migrate off Datadog or PagerDuty.
Connect Struct to your existing stack and run your first agentic on-call investigation today.
What results production teams see with agentic on-call systems
Production teams using agentic on-call systems report consistent, measurable reductions in triage time, engineer-hours spent on incidents, and on-call burnout. The most detailed public proof point comes from Arcana, a Series B fintech company.
After integrating Struct with Sentry, GitHub, GCP Cloud Logging, and Slack, Arcana reduced average developer time per investigation from 30 minutes to 2 minutes, reclaimed 56 engineer-hours per month, and scaled investigation coverage 17x while maintaining an over-80% helpful investigation rate across 2,500+ investigations. Arcana's CTO Siva Sooriyan described the outcome: “We investigate every Sentry alert — no exceptions. Struct made that sustainable by eliminating manual triage. Senior engineers got their time back for higher-leverage work, and the broader team adopted it without friction.”
Broader industry data corroborates these results. Evidence reports MTTR reductions such as 38% for CMC Networks and 33% for HCL after AIOps/event-correlation deployments, with no support for a 2026 analysis or 58% cut in 30 days. Western Governors University reduced incident resolution time from roughly two hours to 28 minutes (approximately 77% MTTR improvement) using the AWS DevOps Agent, per customer stories including Dynatrace.
The compounding effect matters for lean teams. A Series A fintech with 40 engineers running Struct automated their Slack alerting channels and cut triage time by 80%. Newer engineers could then take on-call shifts confidently because Struct provided a structured starting point for every alert, which removed the tribal knowledge bottleneck that previously required senior engineer escalation for every non-trivial incident.
Frequently Asked Questions
What is the difference between an agentic on-call system and a traditional AIOps platform?
Traditional AIOps platforms focus on alert correlation and noise reduction, so they surface fewer, better-grouped alerts but still require a human to investigate each one manually. Agentic on-call systems go further. They autonomously execute the investigation, correlate signals across multiple observability sources, generate a root-cause hypothesis with supporting evidence, and in higher-autonomy configurations, apply and verify a remediation. The key architectural distinction is the presence of a reasoning loop with tool-calling capability and a verification layer that confirms resolution against live data, rather than simply presenting correlated alerts for human review.
How long does it take to set up an agentic on-call system, and what integrations are required?
Setup time varies significantly by vendor. Struct connects to your existing stack, using Slack or PagerDuty as the alert source, GitHub for code context, and Datadog, Sentry, or cloud logs for observability, in minutes with no enterprise deployment or sales process required. The minimum viable integration is an alert source plus one observability tool. Adding GitHub and additional observability sources improves investigation accuracy. Teams with well-structured logging, trace IDs, and existing alerting in Sentry or Datadog see the highest helpful-investigation rates immediately. If your system lacks basic logging or alerting triggers, any agentic system will have limited data to reason over.
What does incident resolution verification actually check, and how is it different from just closing a ticket?
Incident resolution verification queries live observability data such as error rates, latency percentiles, log patterns, and monitor states after a remediation action is applied, then compares the current signal against the pre-incident baseline and the expected post-remediation state. Closing a ticket is a human administrative action. Verification is an automated, evidence-backed confirmation that the underlying condition is cleared. Struct's implementation runs this check approximately every minute during a configurable bake-in window. If metrics regress during that window, the incident remains open and the on-call engineer is notified rather than discovering the re-open through a new customer complaint. This closed-loop approach is the architectural difference between knowing a fix was applied and knowing a fix worked.
Is it safe to give an AI agent autonomous remediation authority in a production environment?
Safety in autonomous remediation depends on blast-radius controls rather than a simple yes or no answer. The standard production architecture classifies actions into read-only, reversible, and irreversible tiers. Reversible actions such as pod restarts, cache clears, feature-flag rollbacks, and scale-out operations can safely reach Level 3 or Level 4 autonomy once they have proven safe across multiple incidents at the approval-gated level. Irreversible actions such as database migrations, IAM changes, and multi-service rollbacks should remain at Level 3 with explicit human approval gates permanently for most teams. Circuit breakers, least-privilege RBAC, dry-run simulation before execution, and automatic rollback on failed verification are the five mandatory guardrails before any action reaches autonomous execution. SOC 2 Type II and HIPAA compliance, as Struct maintains, provide the audit trail and access controls that make this architecture defensible to security and compliance reviewers.
Conclusion: Practical next steps for production teams
Agentic on-call systems reduce the investigation phase of incident response from 30–45 minutes of manual log-hunting to under five minutes of automated correlation, with incident resolution verification closing the loop by confirming fixes against live observability data rather than relying on human spot-checks. Most Series A–C engineering teams are best served by starting at Level 2 or Level 3 autonomy, which means automated investigation with human-approved remediation, then promoting specific runbooks to Level 4 as confidence accumulates.
The architectural requirements are well-established. Teams need a shared knowledge graph, specialized agents for detection, hypothesis generation, validation, planning, and governance, MCP-standardized tool access, A2A coordination for multi-agent workflows, and a verification layer with automatic rollback. The safety requirements are equally clear. Blast-radius limits, least-privilege RBAC, circuit breakers outside the agent control plane, and human approval gates for irreversible actions form the baseline.
For deeper context on related topics, the AI SRE & agentic on-call hub covers the broader category, while the automated root cause analysis spoke details the investigation layer and the incident tracking spoke covers status management and resolution workflows.
Struct implements all of these layers with a fast setup, SOC 2 Type II and HIPAA compliance, and a composable runbook architecture that encodes your team's specific investigation procedures. The Incident Tracker's one-minute verification loop is the practical expression of incident resolution verification for production engineering teams.
Start your Struct pilot and automate your first on-call runbook in under 10 minutes.
