Written by: Nimesh Chakravarthi, Co-founder & CTO, Struct
Key Takeaways for On-Call Teams
- Closed-loop verification uses a Sense-Analyze-Act-Verify cycle that confirms automated fixes restore the expected state with real observability data, unlike open-loop scripts that skip verification.
- Without automated verification, Series A–C engineering teams waste 30–45 minutes investigating alerts that should resolve in two minutes, risking SLA breaches and lost revenue.
- The four-stage loop pulls telemetry from Datadog and Sentry, correlates logs and traces, executes remediation, and re-checks every 60 seconds until baseline conditions are met.
- Strong verification signals query observability metrics directly (latency, error rates, endpoint status) rather than relying on weak indicators like script exit codes.
- Struct automates your on-call runbook with a fast verification loop that confirms every remediation against measurable baselines before closing incidents.
Why Incident Resolution Verification Matters for Series A–C Teams
Incident resolution verification proves that an alert is truly resolved instead of temporarily masked. Teams relying only on script-success checks still spend 30–45 minutes investigating what should have taken two. IBM’s ITBench evaluation tested current AI models against 42 real-world SRE scenarios and found that models resolved only 13.8% of them, underscoring that verification must be tied to actual incident outcomes rather than self-reported success.
An e-commerce platform processing $10,000 per minute loses that revenue for every minute the on-call loop stays open. For Series A–C B2B SaaS engineering teams operating under strict SLAs, every unverified remediation becomes a financial and reliability risk. Struct’s Incident Tracker supplies a 1-minute automated verification loop against Datadog and Sentry data, closing the cycle before an engineer finishes reading the alert.
The Sense-Analyze-Act-Verify loop addresses this gap through four connected stages that run continuously until the system returns to a healthy state. The following sections walk through how each stage works in practice for on-call engineers.
See how Struct closes the verification loop
Sense: Capture Telemetry the Moment Alerts Fire
The Sense stage captures all relevant telemetry as soon as an alert triggers. Production engineering teams connect monitoring, logging, and observability tools as event sources so alerts and diagnostic data flow into the same workflow, enabling normalized events from metrics, logs, traces, and health checks. Datadog metric queries and Sentry error counts provide the raw signals that later stages evaluate.
Struct listens to your designated Slack alerting channels and immediately pulls telemetry from Datadog, Sentry, AWS CloudWatch, and connected observability sources when an alert fires. This approach removes manual log-hunting and avoids context-switching between multiple tabs during a 3 AM incident.
See how Struct pulls telemetry automatically
Analyze: Compare Live Signals to Healthy Baselines
The Analyze stage compares current values against the healthy baseline defined in your runbooks. In the closed-loop model, the system confirms whether the expected healthy state has been restored after remediation, which prevents false confidence and separates successful fixes from failed remediations that require escalation. Struct automatically correlates logs, traces, and code changes within the first minute.
Good playbooks define which diagnostics to run for each incident type, correlate changes with likely causes, and attach collected data to the incident for fast verification, including automated log queries and metric snapshots fired on incident creation. Struct encodes your specific runbook logic so the analysis reflects your system’s architecture, not a generic template.
This analysis output then drives the next stage, where the system or engineer applies the remediation that best matches the diagnosed cause.
Encode your runbook logic with Struct
Act: Apply Remediation Without Leaving Slack
The Act stage executes the remediation that the analysis identified as most likely to restore the baseline. In a bounded DARV cycle, engineers operate “on the loop” rather than “in the loop”: the agent executes the full cycle within guardrails while engineers supervise boundaries, approve promotions, and audit tamper-evident replays instead of hand-typing every command. Struct surfaces the exact command or pull request link directly in Slack so you can approve or trigger the fix without switching tools.
Automated execution of a remediation action takes under five seconds when all stages run sequentially with no human handoff. Once the root cause is confirmed, Struct can hand off context to a coding agent or generate a pull request directly, completing the path from alert to code fix while keeping the workflow inside your existing tools.
Review and trigger fixes from Slack
Verify: Confirm the System Has Truly Recovered
The Verify stage runs targeted checks to confirm the expected state has returned and remains stable. At the advanced stage of automation maturity, the system applies a fix and only escalates if the fix does not hold, using post-remediation observability checks to confirm resolution before closing the incident. Struct’s Incident Tracker performs this verification every 60 seconds until the incident closes.
Closed-loop automation detects a condition, decides on a response, and executes that response without waiting for manual intervention, while still leaving exceptions and novel situations to human oversight. Struct follows the same principle: routine verifications run automatically, and only genuinely novel failure signatures escalate to your on-call engineer.
Now that the four-stage loop is clear, the next step is understanding what separates reliable verification from checks that only confirm a script finished running.
Start automated verification in your runbook
Weak vs. Strong Verification Signals in Practice
Not all verification signals carry equal weight. Script exit codes and absence-of-new-errors checks are weak because they confirm process completion, not state restoration. Strong signals query your observability layer directly and assert a measurable baseline condition over a sustained window.
The table below contrasts weak verification approaches, which only confirm that a script ran, with strong approaches that query actual system health metrics.
| Signal Type | Weak Example | Strong Example |
|---|---|---|
| Datadog | “script exit code 0” | avg(last_5m):avg:service.latency{env:prod} < 200 |
| Sentry | “no new exception logged” | count(last_5m):count:errors{release:current} = 0 |
| Overall | Task completed | Baseline restored for 5 consecutive minutes |
A practical agentic IT management workflow includes a verification stage that checks whether the action produced the expected result, followed by a record stage that preserves the decision, inputs, action, and result for review. Struct’s Incident Tracker applies strong-signal queries at every verification cycle, not just at incident close.
What On-Call Engineers Should Verify After a Fix
Verification only works when it targets the specific signals that define a healthy state for the affected service. The following five checks form a production-grade verification checklist for on-call engineers:
- Service latency returns below threshold for five consecutive minutes.
- Error rate in Sentry drops to zero for the affected release.
- All dependent endpoints return 2xx status codes.
- No new high-severity alerts fire in the same component.
- Customer impact metric (e.g., checkout success rate) recovers to baseline.
The four stages of the closed-loop process include a final verify-and-document step that confirms restoration of the healthy state, with all outcomes preserved via logs, reports, and audit trails. These five checks implement that verify-and-document principle in concrete terms. Struct records every verification result against your Datadog and Sentry data so the audit trail stays complete without extra manual work.
Capture a full verification audit trail with Struct
How Struct’s Incident Tracker Runs the 1-Minute Loop
Struct listens to your Slack alerts, runs the full Sense-Analyze-Act-Verify cycle, and updates incident status automatically. The Incident Tracker, launched August 3, 2026, performs active incident tracking with a ~1-minute automated verification loop against observability data to confirm an incident is actually resolved, not just remediated.
Customer Arcana achieved this improvement in practice, reclaiming 56 engineer-hours per month and running 2,100+ automated investigations monthly. New Relic’s 2026 AI Impact Report, based on aggregated data from 6.6 million platform users, found that AI users achieved 2x higher correlation rates than non-AI accounts, the same category of improvement Struct delivers specifically for incident resolution verification. Struct sits directly on your existing Datadog and Sentry data as an investigation layer and keeps your current observability tooling in place.
Watch Struct run the full incident loop
Frequently Asked Questions
What minimum tooling maturity is required?
You need Datadog or Sentry alerts routed to Slack plus basic trace IDs. Teams below this level cannot yet close the verification loop reliably because Struct’s verification queries depend on structured observability signals. If your system lacks basic logging, trace IDs, or alerting triggers, the automated verification stage cannot assert a measurable healthy state.
How much integration effort is involved?
Teams connect Slack, GitHub, Datadog, and Sentry in under 10 minutes. No code changes, on-prem agents, or lengthy enterprise deployment processes are required. Once connected, auto-investigations start immediately and the Incident Tracker begins its rapid verification loop on the first alert that fires.
What happens when telemetry is limited?
Struct reports exactly which signals are missing and still verifies the signals you do have. It never fabricates data or infers a healthy state from incomplete evidence. When a required metric is unavailable, the verification cycle flags the gap explicitly rather than silently passing the check.
How does Struct handle compliance requirements?
All processing is ephemeral and Struct is SOC 2 Type II plus HIPAA compliant, documented at trust.struct.ai. For organizations with strict rules requiring full on-premises deployment where no logs can leave the internal system, Struct’s current architecture requires cloud-accessible integrations to function.
Review Struct’s fit for your compliance needs
Conclusion: Close Every Incident with Verified Data
Manual verification leaves incidents open and wastes senior engineer time on work that produces no signal beyond “it seems fine.” Self-healing automation fails on novel failure signatures, multi-service coordination requirements, and ambiguous verification signals that produce false closure, which is why production-grade incident resolution verification must rely on real observability data, not process completion flags.
Struct’s Incident Tracker delivers a 1-minute automated verification loop purpose-built for on-call engineering teams at Series A–C B2B SaaS companies. Working with your existing observability stack, Struct encodes your runbooks and confirms every remediation against measurable baseline conditions before closing an incident. For further reading, explore Struct’s hub on Incident Tracking & Resolution Verification, and sibling guides on automated root cause analysis and deploy health monitoring with Deploy Guard.
