{"id":780,"date":"2026-07-12T05:11:02","date_gmt":"2026-07-12T05:11:02","guid":{"rendered":"https:\/\/struct.ai\/articles\/key-sre-principles-on-call\/"},"modified":"2026-08-08T01:16:18","modified_gmt":"2026-08-08T01:16:18","slug":"key-sre-principles-on-call","status":"publish","type":"post","link":"https:\/\/struct.ai\/articles\/key-sre-principles-on-call\/","title":{"rendered":"Key Principles of Site Reliability Engineering for On-Call"},"content":{"rendered":"<p><em>Written by: Nimesh Chakravarthi, Co-founder &amp; CTO, Struct<\/em><\/p>\n<h2>Key Takeaways<\/h2>\n<ul>\n<li>\n<p>SRE on-call acts as a reliability feedback loop that improves software quality when it runs smoothly, but it quickly burns out engineers when it depends on manual log hunting across scattered tools.<\/p>\n<\/li>\n<li>\n<p>Seven core principles create a measurable framework for sustainable on-call: actionable alerting, defined error budgets, toil reduction, bounded pager load, blameless postmortems, runbook-driven response, and continuous feedback loops.<\/p>\n<\/li>\n<li>\n<p>Healthy targets include limiting pages to two per shift, keeping MTTD under five minutes, and holding toil below 50% of on-call time so teams avoid overload and protect engineer health.<\/p>\n<\/li>\n<li>\n<p>Runbooks and automated investigation tools transfer tribal knowledge, cut escalation rates, and let junior engineers handle more incidents on their own.<\/p>\n<\/li>\n<li>\n<p><a target=\"_blank\" rel=\"noopener noreferrer nofollow\" href=\"https:\/\/cal.com\/deepanm\/struct-demo\">Automating your on-call runbook<\/a> with Struct removes the first 30\u201345 minutes of manual triage so engineers can focus on fixing the problem instead of repeating the same investigation steps.<\/p>\n<\/li>\n<\/ul>\n<h2>How SRE On Call Keeps Systems Reliable<\/h2>\n<p><a target=\"_blank\" rel=\"noindex nofollow\" href=\"https:\/\/sre.google\/sre-book\/table-of-contents\/\">Google&#8217;s SRE book<\/a> defines site reliability engineering as applying software engineering practices to operations problems. On-call sits at the center of that work. Engineers carry a pager, respond to alerts, and own production reliability during their shift. The goal extends beyond fixing incidents. Teams use incident data and postmortems to make future incidents less likely and less severe.<\/p>\n<p>The seven principles that make on-call sustainable:<\/p>\n<ol>\n<li>\n<p>Actionable alerting only<\/p>\n<\/li>\n<li>\n<p>Defined error budgets<\/p>\n<\/li>\n<li>\n<p>Toil reduction as a first-class goal<\/p>\n<\/li>\n<li>\n<p>Bounded pager load<\/p>\n<\/li>\n<li>\n<p>Blameless postmortems<\/p>\n<\/li>\n<li>\n<p>Runbook-driven response<\/p>\n<\/li>\n<li>\n<p>Continuous feedback loops<\/p>\n<\/li>\n<\/ol>\n<h2>Principle 1: Actionable Alerting Only<\/h2>\n<p>Every alert that fires must require a human decision. Informational, self-resolving, or consistently ignored alerts create noise, and noise destroys the signal-to-noise ratio that keeps on-call manageable.<\/p>\n<p>Start by auditing every alert rule quarterly and delete or demote any that fired without requiring action in the last 30 days. For alerts that remain, write descriptions that explain the user impact instead of only listing a metric threshold. Route low-severity alerts into a ticket queue instead of a pager to preserve focus for true emergencies. Finally, require a linked runbook for every paging alert before it reaches production so responders know exactly what to do next.<\/p>\n<p><strong>Target metric:<\/strong> <a target=\"_blank\" rel=\"noindex nofollow\" href=\"https:\/\/liuning0820.github.io\/2019\/03\/30\/the-SRE-workbook-reading-notes.html\">Google&#8217;s SRE Workbook<\/a> recommends targeting a maximum of two incidents per on-call shift.<\/p>\n<h2>Principle 2: Defined Error Budgets<\/h2>\n<p>An <a target=\"_blank\" rel=\"noindex nofollow\" href=\"https:\/\/sre.google\/sre-book\/embracing-risk\/\">error budget<\/a> is the acceptable amount of unreliability derived from a service-level objective, or SLO. It turns an abstract reliability target into a concrete operational constraint that guides when to ship features and when to invest in stability.<\/p>\n<p>Define SLOs for every user-facing service before placing it on call. Calculate the error budget as 100% minus the SLO target, expressed in minutes of downtime per month. Freeze non-critical deployments automatically when the error budget drops below 10%. Review budget burn rate weekly in engineering stand-ups, not only after incidents.<\/p>\n<p><strong>Target metric:<\/strong> Error budget burn rate above 2\u00d7 the expected rate for more than one hour reliably signals an impending SLO breach.<\/p>\n<h2>Principle 3: Toil Reduction as a First-Class Goal<\/h2>\n<p><a target=\"_blank\" rel=\"noindex nofollow\" href=\"https:\/\/sre.google\/sre-book\/eliminating-toil\/\">Toil<\/a> is manual, repetitive, automatable work that scales with service growth and produces no lasting value. On-call toil such as acknowledging duplicate alerts, copy-pasting log queries, and manually correlating trace IDs is a primary driver of engineer burnout.<\/p>\n<p>Track toil hours per rotation explicitly and keep less than 50% of on-call time spent on toil. Automate any investigation step performed identically more than three times. Deduplicate alert streams before they reach the pager. Encode repeated diagnostic queries into runbooks or automated tooling so humans do not repeat the same steps by hand.<\/p>\n<p><strong>Target metric:<\/strong> <a target=\"_blank\" rel=\"noindex nofollow\" href=\"https:\/\/cloud.google.com\/blog\/products\/management-tools\/identifying-and-tracking-toil-using-sre-principles\">Google SREs aim for less than half of their time to be spent on toil.<\/a><\/p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer nofollow\" href=\"https:\/\/cal.com\/deepanm\/struct-demo\"><strong>Eliminate repetitive investigation toil<\/strong><\/a> \u2014 Struct encodes your existing runbooks and executes the first-pass investigation automatically the moment an alert fires.<\/p>\n<h2>Principle 4: Bounded Pager Load<\/h2>\n<p>Bounded pager load protects engineers from unsustainable expectations. Without explicit limits, on-call engineers absorb every reliability debt the team has accumulated, regardless of whether that workload is safe.<\/p>\n<p>Set a hard cap of no more than two pages per shift as the team&#8217;s operational target. Track pages per shift as a team health metric in every sprint retrospective. Escalate to engineering leadership when the rolling four-week average exceeds the cap. Rotate on-call schedules to distribute load evenly across the team, including senior engineers.<\/p>\n<p><strong>Target metric:<\/strong> Mean time to detect under five minutes combined with fewer than two pages per shift indicates a well-tuned alerting system.<\/p>\n<h2>Principle 5: Blameless Postmortems<\/h2>\n<p>A <a target=\"_blank\" rel=\"noindex nofollow\" href=\"https:\/\/sre.google\/sre-book\/postmortem-culture\/\">blameless postmortem<\/a> treats every incident as a systems failure, not a human failure. The output is a set of action items that make the system more resilient, not a record of who made a mistake.<\/p>\n<p>Complete a postmortem for every incident that breaches an SLO within 48 hours of resolution. Structure postmortems around a five-why root cause analysis instead of a timeline of blame. Assign every action item an owner and a due date before the postmortem document is closed. Publish postmortems internally so the entire engineering organization learns from each incident.<\/p>\n<p><strong>Target metric:<\/strong> Postmortem action item completion rate above 80% within 30 days shows that the process is driving real reliability improvements.<\/p>\n<h2>Principle 6: Runbook-Driven Response<\/h2>\n<p>A runbook is a documented, step-by-step procedure for diagnosing and resolving a known class of incident. Runbooks transfer tribal knowledge from senior engineers to the entire team. This makes it safe to put junior engineers on call without creating unnecessary escalation risk.<\/p>\n<p>Require a runbook for every paging alert before it reaches production. Version-control runbooks alongside the code they describe. Review and update runbooks after every incident where the documented steps were insufficient. Test runbooks in staging environments before relying on them in production incidents.<\/p>\n<p><strong>Target metric:<\/strong> Escalation rate below 15% per shift indicates runbooks are comprehensive enough for on-call engineers to resolve most incidents independently.<\/p>\n<h2>Principle 7: Continuous Feedback Loops<\/h2>\n<p>On-call data such as pages, MTTD, MTTR, and error budget consumption directly measures system reliability. Teams that feed this data back into sprint planning and architecture decisions improve reliability systematically instead of reacting only after major failures.<\/p>\n<p>Review on-call metrics in every sprint retrospective alongside feature velocity metrics. Treat recurring pages for the same root cause as a P1 engineering priority. Use MTTR trends to identify which services need architectural investment. Share on-call health dashboards with product and leadership so reliability stays visible.<\/p>\n<p><strong>Target metric:<\/strong> Mean time to resolution trending downward quarter-over-quarter confirms the feedback loop is working.<\/p>\n<h2>Making On-Call Sustainable in Practice<\/h2>\n<p>On-call sustainability depends on three layers working together. Alert quality must be high enough that every page is worth waking up for. Investigation tooling must be fast enough that resolution does not require 45 minutes of manual log hunting. Organizational norms must treat pager load as a team metric rather than an individual burden.<\/p>\n<p>The seven principles above address each of these layers. The most common failure mode comes from treating them as aspirational instead of operational. Sustainability requires measuring each principle with a concrete metric and escalating when targets are missed.<\/p>\n<h2>On-Call Health Benchmarks<\/h2>\n<p>The following benchmarks translate these principles into concrete targets. They highlight the difference between healthy on-call operations and teams that are approaching burnout.<\/p>\n<table style=\"min-width: 75px\">\n<colgroup>\n<col style=\"min-width: 25px\">\n<col style=\"min-width: 25px\">\n<col style=\"min-width: 25px\"><\/colgroup>\n<tbody>\n<tr>\n<th colspan=\"1\" rowspan=\"1\">\n<p>Metric<\/p>\n<\/th>\n<th colspan=\"1\" rowspan=\"1\">\n<p>Healthy Target<\/p>\n<\/th>\n<th colspan=\"1\" rowspan=\"1\">\n<p>Warning Threshold<\/p>\n<\/th>\n<\/tr>\n<tr>\n<td colspan=\"1\" rowspan=\"1\">\n<p>Pages per on-call shift<\/p>\n<\/td>\n<td colspan=\"1\" rowspan=\"1\">\n<p><a target=\"_blank\" rel=\"noindex nofollow\" href=\"https:\/\/sre.google\/workbook\/alerting-on-slos\/\">\u2264 2 actionable pages<\/a><\/p>\n<\/td>\n<td colspan=\"1\" rowspan=\"1\">\n<p>&gt; 5 pages per shift<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td colspan=\"1\" rowspan=\"1\">\n<p>Mean Time to Detect (MTTD)<\/p>\n<\/td>\n<td colspan=\"1\" rowspan=\"1\">\n<p>&lt; 5 minutes<\/p>\n<\/td>\n<td colspan=\"1\" rowspan=\"1\">\n<p>&gt; 15 minutes<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td colspan=\"1\" rowspan=\"1\">\n<p>Mean Time to Resolve (MTTR)<\/p>\n<\/td>\n<td colspan=\"1\" rowspan=\"1\">\n<p>&lt; 30 minutes<\/p>\n<\/td>\n<td colspan=\"1\" rowspan=\"1\">\n<p>&gt; 60 minutes<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td colspan=\"1\" rowspan=\"1\">\n<p>Error budget consumption<\/p>\n<\/td>\n<td colspan=\"1\" rowspan=\"1\">\n<p>&lt; 50% consumed mid-month<\/p>\n<\/td>\n<td colspan=\"1\" rowspan=\"1\">\n<p>&gt; 90% consumed before month-end<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td colspan=\"1\" rowspan=\"1\">\n<p>Toil as % of on-call hours<\/p>\n<\/td>\n<td colspan=\"1\" rowspan=\"1\">\n<p><a target=\"_blank\" rel=\"noindex nofollow\" href=\"https:\/\/sre.google\/sre-book\/eliminating-toil\/\">&lt; 50%<\/a><\/p>\n<\/td>\n<td colspan=\"1\" rowspan=\"1\">\n<p>&gt; 70%<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><a target=\"_blank\" rel=\"noopener noreferrer nofollow\" href=\"https:\/\/cal.com\/deepanm\/struct-demo\"><strong>See how Struct hits these MTTR targets<\/strong><\/a> \u2014 Move your team\u2019s resolution time from 45 minutes to under 5 in a 10-minute live demo.<\/p>\n<h2>Where Automated Investigation Fits<\/h2>\n<p>Most incidents begin with 30 to 45 minutes spent on tasks that do not require human judgment. Engineers acknowledge the alert, pull logs from CloudWatch or Datadog, find the relevant exception in Sentry, cross-reference the recent deploy in GitHub, and assemble a timeline. This work is pure toil and keeps MTTR high even on teams that follow the seven principles above.<\/p>\n<p>Struct removes this phase entirely. When an alert fires in a designated Slack channel or PagerDuty queue, Struct automatically queries every connected observability source such as Datadog, AWS CloudWatch, GCP Logs, Azure Traces, Sentry, Grafana, and GitHub. It correlates the signals into a unified timeline, identifies the likely root cause, and delivers a dynamically generated dashboard to the alert thread before the on-call engineer opens their laptop. The investigation finishes in under five minutes. The engineer reviews the evidence, confirms the root cause, and moves directly to resolution.<\/p>\n<p>Teams with custom runbooks can encode those procedures directly so every automated investigation follows the same diagnostic steps a senior engineer would take. Junior engineers on call receive a fully contextualized starting point for every alert, which reduces escalation rates and makes it safe to distribute on-call load across the full team. The result is an 80% reduction in triage time and MTTR that trends toward the healthy benchmark in the table above from the first shift.<\/p>\n<h2>Conclusion<\/h2>\n<p>Automated investigation closes the execution gap that prevents teams from reaching these benchmarks manually. These principles form a complete framework for turning on-call from a burnout rotation into a reliability engine when teams apply them together. Each principle is measurable, and the benchmarks table above provides concrete targets. Struct removes the triage phase described earlier so engineers can apply this framework at scale instead of only in theory.<\/p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer nofollow\" href=\"https:\/\/cal.com\/deepanm\/struct-demo\"><strong>Start automating your incident response<\/strong><\/a> \u2014 Set up Struct in under 10 minutes and let AI handle your next on-call investigation from alert to root cause before you finish your coffee.<\/p>\n<h2>Frequently Asked Questions<\/h2>\n<h3>What is the difference between MTTD and MTTR in SRE on-call?<\/h3>\n<p>Mean Time to Detect measures how long it takes from when a problem begins to when an alert fires and reaches the on-call engineer. Mean Time to Resolve measures how long it takes from detection to full resolution. MTTD primarily depends on alerting quality, including thresholds, coverage, and routing. MTTR primarily depends on investigation speed and runbook quality. Teams track both metrics separately because they have different root causes and different remediation strategies. A team can have excellent MTTD but poor MTTR when alerting is well tuned but the investigation process remains manual and slow.<\/p>\n<h3>How many pages per shift is considered healthy for an SRE on-call rotation?<\/h3>\n<p>Google&#8217;s SRE practice focuses on keeping on-call pager load at a sustainable level so engineers have time to investigate, resolve, and document each incident. When page volume stays high over time, teams should audit their alert rules, deduplicate noisy sources, and automate the investigation of recurring alert classes before adding headcount to absorb the load.<\/p>\n<h3>How does an error budget connect to on-call decision-making?<\/h3>\n<p>An error budget is the quantified tolerance for unreliability derived from a service-level objective. In on-call practice, it functions as a real-time decision framework. When the error budget is healthy, the team has operational runway to ship features and accept deployment risk. When the budget is burning faster than expected or is nearly exhausted, the team should freeze non-critical changes, prioritize reliability work, and increase scrutiny on deployments. Tracking error budget burn rate during on-call shifts gives engineers an objective basis for escalating reliability concerns to product and leadership rather than absorbing the risk silently.<\/p>\n<h3>Can junior engineers safely take on-call shifts without deep system knowledge?<\/h3>\n<p>Junior engineers can take on-call shifts safely when the right tooling and runbooks are in place. The primary barrier for junior engineers on call is the lack of tribal knowledge required to navigate complex, multi-service architectures during an incident. Comprehensive runbooks that document diagnostic steps for every paging alert class significantly lower this barrier. Automated investigation tools that perform the first-pass triage by correlating logs, identifying the blast radius, and surfacing a root cause hypothesis lower it further by giving junior engineers a fully contextualized starting point instead of a blank screen. Teams that combine detailed runbooks with automated investigation tooling report materially lower escalation rates and faster onboarding of new on-call engineers.<\/p>\n<h3>What is toil in SRE and why does it matter for on-call sustainability?<\/h3>\n<p>Toil is manual, repetitive, automatable work that scales linearly with service growth and produces no lasting improvement to system reliability. In on-call rotations, toil includes tasks such as acknowledging duplicate alerts, manually pulling the same log queries for recurring incidents, copy-pasting correlation IDs across observability tools, and re-running the same diagnostic steps documented in a runbook. Toil matters because it consumes engineering capacity without generating reliability improvements. Google&#8217;s SRE practice sets a target of less than 50% of on-call hours spent on toil, and teams above that threshold are effectively paying senior engineers to perform work that automation can handle, which reduces product velocity and accelerates burnout.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Master the 7 SRE on-call principles\u2014actionable alerting to blameless postmortems. Struct automates triage so your team focuses on fixing, not finding.<\/p>\n","protected":false},"author":73,"featured_media":793,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-780","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/posts\/780","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/comments?post=780"}],"version-history":[{"count":1,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/posts\/780\/revisions"}],"predecessor-version":[{"id":795,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/posts\/780\/revisions\/795"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/media\/793"}],"wp:attachment":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/media?parent=780"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/categories?post=780"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/tags?post=780"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}