Written by: Nimesh Chakravarthi, Co-founder & CTO, Struct
Key Takeaways
- Healthy on-call programs target no more than 2 actionable pages per engineer per shift, a false-positive rate below 10%, and MTTR under 15 minutes.
- A seven-step playbook starts with a baseline audit of pages, false positives, MTTR, and runbook coverage before any changes happen.
- Core practices include SLO-based alerting, a “no runbook, no page” policy, AI-assisted first-pass investigation, equitable rotations, blameless postmortems, a pager health dashboard, and quarterly maturity reviews.
- Teams move through four maturity stages, from manual triage to full AI-assisted remediation, while keeping existing tools and workflows.
- Struct helps teams reduce triage time by 80%, compressing a 45-minute manual investigation to under 5 minutes for Seed to Series C companies. See how Struct compresses triage time and get your first AI-generated investigation running in under 10 minutes.
Section 1 – Define the Objective and Current State
Start by auditing four baseline numbers: pages per shift, false-positive percentage, MTTR, and runbook coverage. Use the checklist below to capture your current state in a single sitting.
| Metric | How to Measure | Healthy Target | Owner |
|---|---|---|---|
| Pages per shift | PagerDuty or alerting tool export, 30-day rolling | ≤2 actionable | On-call lead |
| False-positive rate | (Non-actionable pages ÷ total pages) × 100 | <10% | On-call lead |
| MTTR | Incident tool: time-to-acknowledge to time-to-resolve | <15 min | Engineering manager |
| Runbook coverage | Count alerts with linked runbooks ÷ total alerts | 100% | SRE team |
Any metric outside its target becomes a prioritized input to the seven steps below. These steps address the root causes behind poor on-call metrics. Steps 1 and 2 reduce false positives and MTTR by improving alert quality. Step 3 accelerates triage through automation. Steps 4 and 5 prevent burnout and capture systemic improvements. Steps 6 and 7 ensure the gains persist over time.
Section 2 – 7-Step On-Call Scaling Playbook
1. Set SLO-Based, Actionable-Only Alerting Rules
The goal is to page engineers only when an SLO burn rate threatens an error budget within a defined window. The SRE lead owns this work and starts from defined SLOs per service and historical error-rate data. From those inputs, the team creates alert rules tied to multi-window burn-rate thresholds, such as paging when 2% of the error budget is consumed in 1 hour. This approach requires upfront SLO definition, so teams without SLOs must define them first or use a proxy metric such as 5xx rate above a rolling baseline.
2. Enforce a “No Runbook, No Page” Alert Policy
The goal is to eliminate alerts that fire without a documented response path. The engineering manager and alert author share ownership and begin with an alert inventory and the existing runbook repository. They then add a gate in CI or CD that blocks alert deployment if no runbook URL is attached. This policy slows initial alert creation, and that friction is intentional because it reduces noise and confusion later.
3. Automate First-Pass Investigation with AI Tooling
The goal is to deliver a correlated root-cause summary before the on-call engineer opens their laptop. The SRE lead and tooling owner manage this step using the alert channel, observability integrations, and runbook instructions as inputs. The AI layer produces an auto-generated timeline, blast-radius summary, and suggested fix within 5 minutes of the alert firing. This approach depends on clean telemetry, so teams with sparse logging will see lower accuracy.
The 80% triage-time reduction mentioned earlier translates to compressing a 45-minute manual investigation into under 5 minutes for Seed to Series C companies. Get your first AI investigation running and see this workflow in your own alert channel.
4. Design Fair Rotations and Clear Compensation
The goal is to distribute on-call load fairly and compensate engineers for after-hours work. The engineering manager owns this step and uses team size, time zones, and pages-per-shift data to design the schedule. The output is a rotation with at least 5 engineers per pool, a clear compensation policy using cash, comp time, or both, and a shadow rotation that helps onboard junior engineers. Smaller teams with fewer than 5 engineers face unavoidable concentration risk, so they should first reduce alert volume.
5. Run Blameless Post-Incident Reviews with Owners
The goal is to turn every significant incident into a systemic improvement with a named owner and due date. The incident commander and engineering manager lead this work using the incident timeline, contributing factors, and action items as inputs. They publish a written postmortem within 48 hours, track action items in the team’s ticketing system, and schedule a 30-day follow-up review. This process requires protected calendar time, or action items stall and the practice loses credibility.
6. Create a Shared Pager Health Dashboard
Track the six KPIs below on a shared dashboard that the engineering manager and on-call lead review monthly. This view keeps the team aligned on pager load, quality, and follow-through.
| KPI | Definition | Target | Review Cadence |
|---|---|---|---|
| Pages per shift | Actionable pages per engineer per 8-hour shift | ≤2 | Monthly |
| False-positive rate | Non-actionable pages ÷ total pages | <10% | Monthly |
| MTTR | Acknowledge to resolve, median | <15 min | Monthly |
| Runbook coverage | Alerts with linked runbooks ÷ total alerts | 100% | Monthly |
| Postmortem completion rate | Postmortems published within 48 h ÷ P1/P2 incidents | 100% | Monthly |
| Action-item closure rate | Postmortem action items closed on time ÷ total | ≥80% | Quarterly |
7. Hold a Quarterly On-Call Maturity Review
The goal is to prevent metric drift and move the team along the maturity roadmap. The engineering manager and SRE lead own this review and use the pager health dashboard, postmortem backlog, and error-budget reports as inputs. They leave the session with updated alert rules, rotation adjustments, and a prioritized improvement list for the next quarter. Quarterly reviews form the minimum effective cadence, while monthly reviews work better during rapid growth.
Section 3 – How This Playbook Fits Daily Engineering Work
The seven steps above map directly onto tools engineering teams already use. Alerts originate in observability platforms and route into the team’s communication hub and incident-management system. Runbooks live in the source-control repository alongside the services they describe, which keeps them versioned with the code. Post-incident action items land in the existing ticketing system so they compete for sprint capacity alongside feature work.
| Workflow Stage | Tool Category | Integration Point | Output |
|---|---|---|---|
| Alert fire | Observability platform | Webhook to incident channel | Structured alert payload |
| First-pass investigation | AI triage layer | Reads logs, traces, code, posts summary | Root-cause report in channel |
| Escalation & coordination | Incident management | Auto-creates incident record | Timeline, responder assignment |
| Resolution & follow-up | Ticketing + source control | Action items linked to postmortem | PR or task with owner and due date |
No new communication channels are required. The AI triage layer operates inside the existing alert thread, so engineers receive context without switching tools. Once this workflow lives inside current tools, the next step is to measure whether it actually improves on-call health.
Section 4 – Measurement and Continuous Improvement
The six KPIs from the pager health dashboard drive two review loops. A monthly dashboard review catches metric drift early, and a quarterly roadmap update advances the maturity stage. Engineering managers own the monthly review. The quarterly update requires sign-off from the VP of Engineering or equivalent so improvement work receives sprint allocation.
Start capturing triage-time data from day one and give your pager health dashboard a reliable baseline.
Section 5 – Common Pitfalls and Practical Fixes
| Pitfall | Symptom | Mitigation |
|---|---|---|
| Alert overload | False-positive rate >30%, engineers silence alerts | Enforce SLO-based burn-rate thresholds, delete alerts with no action in 90 days |
| Missing runbooks | MTTR spikes on unfamiliar services | CI gate blocks alert deployment without runbook URL |
| Tribal knowledge concentration | Junior engineers escalate every page to one senior | AI first-pass investigation provides context, shadow rotations transfer knowledge |
| Weak escalation paths | Incidents stall when primary responder is unavailable | Define and test secondary and tertiary escalation paths in the rotation tool |
| Engineer burnout | Voluntary attrition spikes, sick days cluster after on-call weeks | Cap pages per shift at ≤2, compensate explicitly, keep rotation pool size at 5 or more |
2026 On-Call Maturity Roadmap
| Stage | Description | Target Metrics | Key Enabler |
|---|---|---|---|
| Stage 0 – Manual Triage | Engineers hunt logs across tools with no runbooks | MTTR >45 min, false positives >40% | Baseline audit (Section 1) |
| Stage 1 – Structured Response | SLO-based alerts, full runbook coverage, blameless postmortems running | MTTR <30 min, false positives <20% | Steps 1–2 and 5 |
| Stage 2 – AI-Assisted Triage | Automated first-pass investigation, pager health dashboard live | MTTR <15 min, false positives <10%, triage time −80% | Steps 3 and 6, AI investigation layer |
| Stage 3 – Full AI-Assisted Remediation | AI suggests and drafts fixes, error-budget policy gates releases, chaos engineering validates resilience | MTTR <10 min, pages per shift ≤1, junior engineers self-sufficient | Steps 4 and 7, code-agent handoff |
Frequently Asked Questions
What is the minimum team size or tooling maturity needed to start this playbook?
Teams as small as three to five engineers can implement Steps 1, 2, and 5 immediately using any standard alerting and incident-management tool. AI-assisted triage in Step 3 requires existing observability instrumentation, at minimum structured logs with trace IDs and an alert channel in Slack or a ticketing system. Teams without basic logging should instrument services first, because the accuracy of automated investigation depends directly on telemetry quality.
How much engineering time does integration actually take?
Connecting an AI investigation layer to an existing Slack alerting channel, a code repository, and an observability platform takes under 10 minutes for teams already using those tools. The setup time mentioned earlier, under 10 minutes, applies to teams that already rely on Slack, a code repository, and an observability platform. The first automated investigation runs immediately after connection and provides a working baseline for the pager health dashboard from day one.
What happens if our logging and telemetry are poor quality?
AI-assisted triage relies on the data available in connected systems. Sparse logs, missing trace IDs, or unstructured log formats reduce the accuracy of automated root-cause analysis. The practical mitigation is to treat telemetry quality as a prerequisite. Add structured logging and trace propagation to the highest-traffic services first, then expand AI triage coverage as instrumentation improves. Teams with poor telemetry still benefit from Steps 1, 2, 4, and 5, which do not depend on AI tooling.
How do junior engineers safely take on-call shifts without deep system knowledge?
Two mechanisms make junior on-call safe. A shadow rotation pairs juniors with a senior for two to four weeks before independent shifts and uses real incidents as training material. AI first-pass investigation then provides a contextualized starting point, including blast radius, correlated timeline, and suggested fix, for every alert. Juniors can follow the AI-generated summary and escalate only when the suggested path does not resolve the issue, instead of escalating by default.
Is an AI investigation tool compliant with strict security and data requirements?
Struct is SOC 2 and HIPAA compliant, which covers the compliance requirements of most Series A–C companies in the United States. Logs and telemetry data are accessed and processed ephemerally, and they are not stored beyond the investigation window. Teams with enterprise policies that require full on-premise deployment or zero-egress log access should confirm that a cloud-integrated tool fits their security posture before adoption.
Conclusion
Scaling on-call reliability without adding headcount requires a clean baseline audit, SLO-based alerting that removes noise at the source, AI-assisted first-pass investigation that compresses triage from 45 minutes to under 5, and a quarterly review cadence that prevents metric drift. The seven steps above provide a concrete, owner-assigned path from Stage 0 manual triage to Stage 3 AI-assisted remediation. The next frontier beyond this playbook couples the maturity roadmap with a formal error-budget policy that gates feature releases on reliability targets and validates resilience assumptions through structured chaos engineering experiments.
Connect your integrations in under 10 minutes and let AI handle the next investigation before your engineer opens their laptop.
