{"id":636,"date":"2026-06-15T05:47:32","date_gmt":"2026-06-15T05:47:32","guid":{"rendered":"https:\/\/struct.ai\/articles\/7-qc-tools-root-cause\/"},"modified":"2026-06-15T05:47:32","modified_gmt":"2026-06-15T05:47:32","slug":"7-qc-tools-root-cause","status":"publish","type":"post","link":"https:\/\/struct.ai\/articles\/7-qc-tools-root-cause\/","title":{"rendered":"7 QC Tools for Root Cause Analysis in Production Incidents"},"content":{"rendered":"<p><em>Written by: Nimesh Chakravarthi, Co-founder &amp; CTO, Struct<\/em><\/p>\n<h2 id=\"key-takeaways\">Key Takeaways<\/h2>\n<ul>\n<li>The 7 QC tools create a repeatable workflow that fits within 30 minutes and maps directly to logs, traces, and deployment events.<\/li>\n<li>Running check sheet, Pareto, fishbone, histogram, scatter, flowchart, and control-chart analysis in sequence turns scattered telemetry into a confirmed root cause with evidence.<\/li>\n<li>A worked 3 AM latency-spike example shows the full sequence finishing in 20 minutes and isolating an ORM cache-key change as the cause.<\/li>\n<li>Manual data assembly, such as querying logs, correlating trace IDs, and reconstructing timelines, remains the largest time sink. The Skip-vs-Use table helps teams apply only the tools their incident profile requires.<\/li>\n<li>Struct automates the data-assembly phase so engineers arrive at analysis instead of hunting. <a href=\"https:\/\/cal.com\/deepanm\/struct-demo\" target=\"_blank\">See how Struct pre-assembles your incident data<\/a>.<\/li>\n<\/ul>\n<h2>Why Root Cause Analysis Breaks at 3 AM<\/h2>\n<p>Late-night incidents punish ad-hoc investigation. When a 3 AM alert fires, the first 30 minutes decide whether you resolve the issue quickly or spend hours chasing guesses. Most engineers start by checking recent deploys, scanning logs, and restarting services. That approach relies on memory and intuition instead of a clear framework.<\/p>\n<p>The 7 QC tools provide that framework. They give you a fixed sequence that turns noisy telemetry into a documented root cause in under 30 minutes. This guide adapts the classic 7 QC tools to software incidents. Each tool maps to a concrete on-call action, a software-specific template, and a worked latency-spike example. Any engineer can run this workflow and see exactly where automation can remove the remaining manual work.<\/p>\n<h2>1. Check Sheet: Structure the Raw Errors<\/h2>\n<p><strong>Purpose:<\/strong> Systematically count and categorize error occurrences so later analysis starts from structured data instead of memory.<\/p>\n<p><strong>How to Use:<\/strong> Before using any other tool, define the error categories relevant to the alert type, such as HTTP 5xx codes, timeout classes, null-pointer exceptions, and database connection failures. These categories become the rows in your check sheet. Next, tally occurrences for each category across a fixed time window, such as the 15 minutes around the alert, so your counts reflect the incident\u2019s impact instead of background noise.<\/p>\n<p><strong>Software Example:<\/strong> Query your log aggregator for the alert window and group log lines by error type. A check sheet for a latency spike might show: DB_TIMEOUT: 412, UPSTREAM_503: 87, NULL_POINTER: 14, OTHER: 9.<\/p>\n<p><strong>Quick Template:<\/strong> Columns: Error Type | Count | First Seen | Last Seen | Affected Service. Rows: one per distinct error class observed in the window.<\/p>\n<h2>2. Pareto Chart: Focus on the Vital Few Errors<\/h2>\n<p>With error counts now structured in the check sheet, the next step is prioritization. A Pareto chart takes those counts and ranks them so you focus on the small set of errors that drive most of the impact.<\/p>\n<p><strong>Purpose:<\/strong> Rank error categories by frequency to identify the vital few causes that drive the majority of impact. <a href=\"https:\/\/monday.com\/blog\/project-management\/pareto-principle\" target=\"_blank\" rel=\"noindex nofollow\">Software teams routinely find that 80% of system crashes originate from just 20% of bugs<\/a>, so Pareto analysis becomes a fast prioritization tool.<\/p>\n<p><strong>How to Use:<\/strong> Sort check-sheet totals in descending order. Plot a bar chart with a cumulative percentage line. <a href=\"https:\/\/domo.com\/learn\/charts\/pareto-charts\" target=\"_blank\" rel=\"noindex nofollow\">Draw an optional 80% reference line to mark the cutoff between high-impact and low-impact items<\/a>. Focus subsequent investigation on categories left of that line.<\/p>\n<p><strong>Software Example:<\/strong> DB_TIMEOUT, with 412 hits and 79% of total errors, sits entirely left of the 80% line. UPSTREAM_503 and NULL_POINTER contribute far less. The investigation narrows to the database layer. <a href=\"https:\/\/netscout.com\/what-is\/root-cause-analysis\" target=\"_blank\" rel=\"noindex nofollow\">In ITSM environments, Pareto analysis of helpdesk tickets has revealed that 80% of system lockouts trace to just two legacy applications<\/a>. The same concentration pattern appears in microservice error distributions.<\/p>\n<p><strong>Quick Template:<\/strong> Columns: Rank | Error Type | Count | % of Total | Cumulative %. Sort descending by Count.<\/p>\n<h2>3. Fishbone Diagram: Map Possible Causes for the Top Error<\/h2>\n<p>The Pareto chart points to the dominant error category. The fishbone diagram then explores why that category spiked before you commit to a single hypothesis.<\/p>\n<p><strong>Purpose:<\/strong> Map all plausible causal factors for the dominant error category before locking in on one explanation.<\/p>\n<p><strong>How to Use:<\/strong> Place the confirmed problem, such as \u201cDB_TIMEOUT spike,\u201d at the diagram head. Branch into four categories suited to software incidents: <a href=\"https:\/\/netscout.com\/what-is\/root-cause-analysis\" target=\"_blank\" rel=\"noindex nofollow\">People (training deficits, fatigue), Process (change management failures, CI\/CD pipeline gaps), Technology (misconfigured connection pools, unpatched drivers), and Data\/Information (corrupt records, missing indexes)<\/a>. Populate each branch with evidence from logs and recent deployment history so every candidate cause ties back to observed data.<\/p>\n<p><strong>Software Example:<\/strong> Under Technology, you might add \u201cconnection pool max_size unchanged at 10 despite traffic doubling.\u201d Under Process, you might add \u201cno load test run before last deploy.\u201d <a href=\"https:\/\/rzsoftware.com\/blog\/fishbone-diagram-root-cause-analysis\" target=\"_blank\" rel=\"noindex nofollow\">Fishbone diagrams work better than 5 Whys alone for complex or systemic issues with multiple interrelated factors<\/a>.<\/p>\n<p><strong>Quick Template:<\/strong> Four branches: People | Process | Technology | Data. Each branch: 2\u20134 candidate causes drawn directly from log and deploy evidence.<\/p>\n<h2>4. Histogram: See the Shape of Latency<\/h2>\n<p>The fishbone diagram lists candidate causes. A histogram then checks whether the latency pattern matches those ideas, such as a split between cached and uncached requests.<\/p>\n<p><strong>Purpose:<\/strong> Reveal the shape of a metric\u2019s distribution, not just its average, so you can distinguish a uniform slowdown from a bimodal or long-tail pattern.<\/p>\n<p><strong>How to Use:<\/strong> Pull p50, p75, p90, p95, and p99 latency values for the affected endpoint across the incident window. Bin them into 50 ms or 100 ms intervals and plot frequency. A bimodal histogram, with two peaks, suggests two distinct request populations, often cached versus uncached or two separate code paths.<\/p>\n<p><strong>Software Example:<\/strong> The \/checkout endpoint shows one cluster at 120 ms and a second cluster at 4,200 ms. The long tail maps to requests that hit the database instead of the cache, which confirms the Technology branch from the fishbone.<\/p>\n<p><strong>Quick Template:<\/strong> X-axis: latency buckets (ms). Y-axis: request count. Annotate p99 and the SLA threshold as vertical lines.<\/p>\n<h2>5. Scatter Diagram: Test Correlations Behind the Incident<\/h2>\n<p>Once the latency shape is clear, a scatter diagram tests whether a suspected driver, such as deploy size, actually correlates with the impact.<\/p>\n<p><strong>Purpose:<\/strong> Test whether a suspected relationship between two variables is real, for example whether deployment size correlates with error rate.<\/p>\n<p><strong>How to Use:<\/strong> Plot one variable on each axis across multiple data points, such as deploys, time windows, or services. A positive correlation strengthens a causal hypothesis. No correlation weakens or rules out that hypothesis. <a href=\"https:\/\/indeed.com\/career-advice\/career-development\/quality-tools\" target=\"_blank\" rel=\"noindex nofollow\">Scatter diagrams are the recommended tool when the goal is to understand relationships between variables<\/a>.<\/p>\n<p><strong>Software Example:<\/strong> X-axis: lines of code changed per deploy. Y-axis: p99 latency in the hour after each deploy. A clear positive correlation across the last 20 deploys confirms that larger changesets predict latency regressions more than time of day or traffic volume.<\/p>\n<p><strong>Quick Template:<\/strong> Columns in the dataset: Deploy ID | Changeset Size (LOC) | Post-Deploy p99 (ms) | Error Rate (%). Plot LOC vs. p99.<\/p>\n<h2>6. Flowchart: Rebuild the Incident Timeline<\/h2>\n<p>After you understand likely causes and correlations, a flowchart reconstructs the exact sequence of events and exposes process gaps.<\/p>\n<p><strong>Purpose:<\/strong> Reconstruct the sequence from the triggering commit to the customer-facing alert so you can see gaps in process or monitoring coverage.<\/p>\n<p><strong>How to Use:<\/strong> Map the incident timeline as a linear flow: Commit \u2192 CI\/CD pipeline \u2192 Deploy \u2192 First anomalous metric \u2192 First log error \u2192 Alert fires \u2192 Engineer paged. Annotate each step with the actual timestamp. Gaps between steps show where detection latency occurs.<\/p>\n<p><strong>Software Example:<\/strong> The deploy completed at 3:02 AM. The first DB_TIMEOUT log appeared at 3:04 AM. The alert fired at 3:17 AM, which creates a 13-minute detection gap caused by a 15-minute evaluation window on the Datadog monitor. The flowchart makes this gap visible in seconds.<\/p>\n<p><strong>Quick Template:<\/strong> Boxes: Commit SHA | Deploy Timestamp | First Error Log | Alert Threshold Crossed | PagerDuty Page | Engineer Acknowledges. Arrows annotated with elapsed time.<\/p>\n<h2>7. Control Chart: Separate Signal from Noise<\/h2>\n<p>The flowchart explains timing. A control chart then confirms that the incident represents a true process break, not normal variation.<\/p>\n<p><strong>Purpose:<\/strong> Distinguish a statistically significant deviation from normal process variation so you avoid both false escalations and missed incidents.<\/p>\n<p><strong>How to Use:<\/strong> Plot the metric, such as p99 latency, over a rolling 7-day baseline. Calculate the upper control limit, or UCL, as the mean plus three standard deviations. Treat any point above the UCL as a signal instead of noise. <a href=\"https:\/\/indeed.com\/career-advice\/career-development\/quality-tools\" target=\"_blank\" rel=\"noindex nofollow\">Control charts are the recommended tool for ongoing monitoring and longer-term improvement<\/a>.<\/p>\n<p><strong>Software Example:<\/strong> The 7-day mean p99 for \/checkout is 130 ms with a standard deviation of 40 ms. UCL equals 250 ms. The 3:17 AM reading of 4,200 ms sits roughly 100 standard deviations above the mean, which makes it an unambiguous signal rather than a traffic fluctuation. This data point also anchors the post-incident SLA report.<\/p>\n<p><strong>Quick Template:<\/strong> X-axis: time, usually hourly. Y-axis: p99 latency (ms). Three horizontal lines: LCL | Mean | UCL. Annotate the incident point.<\/p>\n<h2>Worked Example: Latency Spike at 3:17 AM<\/h2>\n<p><strong>Incident:<\/strong> PagerDuty fires. The \/checkout endpoint p99 latency exceeds 4,000 ms for two consecutive evaluation windows.<\/p>\n<p><strong>Check Sheet (2 min):<\/strong> A log query returns DB_TIMEOUT: 412, UPSTREAM_503: 87, NULL_POINTER: 14. Data is now structured, so analysis can begin.<\/p>\n<p><strong>Pareto Chart (3 min):<\/strong> DB_TIMEOUT represents 79% of all errors. Investigation scope narrows to the database layer.<\/p>\n<p><strong>Fishbone (5 min):<\/strong> Technology branch lists a connection pool capped at 10 connections while traffic doubled after a midnight marketing campaign. Process branch lists no load test before the 2:58 AM deploy that updated the ORM configuration.<\/p>\n<p><strong>Histogram (3 min):<\/strong> Latency distribution is bimodal, with 120 ms for cached requests and 4,200 ms for uncached requests hitting the saturated connection pool. Cache hit rate drops from 91% to 34% after the ORM change alters query key formats.<\/p>\n<p><strong>Scatter Diagram (3 min):<\/strong> Plotting the last 20 deploys shows a strong positive correlation between ORM configuration changes and post-deploy cache miss rate. This deploy appears as a clear outlier.<\/p>\n<p><strong>Flowchart (2 min):<\/strong> Commit at 2:51 AM \u2192 Deploy complete 2:58 AM \u2192 First DB_TIMEOUT log 3:04 AM \u2192 UCL breach 3:17 AM \u2192 Page sent 3:17 AM. The 13-minute detection gap is documented for the post-mortem.<\/p>\n<p><strong>Control Chart (2 min):<\/strong> The 4,200 ms reading is confirmed as a statistically significant signal. The UCL of 250 ms becomes the threshold for the updated automated alert rule.<\/p>\n<p><strong>Total elapsed time: 20 minutes.<\/strong> Root cause: an ORM cache key format change invalidated 57% of the cache and saturated a connection pool that was never sized for full database load. Fix: revert the ORM change, raise connection pool max_size to 50, and restore cache hit rate.<\/p>\n<h2>Skip vs. Use Decision Table for the 7 QC Tools<\/h2>\n<table>\n<thead>\n<tr>\n<th>Tool<\/th>\n<th>Use When<\/th>\n<th>Skip When<\/th>\n<th>Software Signal to Decide<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Check Sheet<\/td>\n<td>Multiple error types appear in logs<\/td>\n<td>A single error type dominates immediately<\/td>\n<td>Log query returns more than two distinct error classes<\/td>\n<\/tr>\n<tr>\n<td>Pareto Chart<\/td>\n<td>You need to prioritize among three or more error categories<\/td>\n<td>One category already exceeds 90% of volume<\/td>\n<td>Check sheet shows distributed error counts<\/td>\n<\/tr>\n<tr>\n<td>Fishbone<\/td>\n<td>Root cause is unknown and several hypotheses exist<\/td>\n<td>Root cause is already confirmed by a prior post-mortem<\/td>\n<td>No single obvious cause appears in logs or recent deploys<\/td>\n<\/tr>\n<tr>\n<td>Histogram<\/td>\n<td>Latency or error rate distribution shape matters<\/td>\n<td>The incident is binary, such as service up or down<\/td>\n<td>Metric has variance and p50 and p99 diverge significantly<\/td>\n<\/tr>\n<tr>\n<td>Scatter Diagram<\/td>\n<td>You are testing a correlation hypothesis, such as deploy size versus error rate<\/td>\n<td>Only one data point exists<\/td>\n<td>At least ten comparable historical events are available<\/td>\n<\/tr>\n<tr>\n<td>Flowchart<\/td>\n<td>You need to document detection latency or process gaps<\/td>\n<td>The incident timeline is already fully logged automatically<\/td>\n<td>Gap between first error log and alert fire time exceeds five minutes<\/td>\n<\/tr>\n<tr>\n<td>Control Chart<\/td>\n<td>You must distinguish signal from noise in a recurring metric<\/td>\n<td>This is the first occurrence of a brand-new metric with no baseline<\/td>\n<td>Seven or more days of historical data exist for the metric<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>When Manual 7 QC Tools Stop Being Enough<\/h2>\n<p>The 20-minute worked example assumes the engineer starts with a clean check sheet, pre-queried logs, and a structured dataset. In real incidents, the first 20 to 25 minutes often disappear into assembling that raw material. Engineers write log queries, correlate trace IDs across Datadog and Sentry, cross-reference GitHub commits, and rebuild a timeline from scattered Slack messages. <a href=\"https:\/\/safetyculture.com\/topics\/root-cause-analysis\" target=\"_blank\" rel=\"noindex nofollow\">Effective root cause analysis requires accurate data collection before any analysis begins<\/a>, and that collection step breaks down when everything is manual.<\/p>\n<p><a href=\"https:\/\/tricentis.com\/learn\/quality-management\" target=\"_blank\" rel=\"noindex nofollow\">Modern software development generates enormous volumes of telemetry, test results, and performance data, and without a clear framework for prioritizing and interpreting that data, teams drown in dashboards instead of acting on insights<\/a>. The 7 QC tools supply the framework. Automation supplies the data, already structured and ready for analysis.<\/p>\n<p>Struct removes the data-assembly phase. When an alert fires in a connected Slack channel or PagerDuty queue, Struct queries logs across Datadog, AWS CloudWatch, GCP, Sentry, and GitHub, correlates trace IDs, builds the incident timeline, and delivers a root cause assessment with suggested fixes before the engineer opens a laptop. The triage window drops from 30\u201345 minutes to 5\u201310 minutes.<\/p>\n<p><a href=\"https:\/\/cal.com\/deepanm\/struct-demo\" target=\"_blank\">Book a demo to see the triage time reduction in action<\/a><\/p>\n<h2>Frequently Asked Questions<\/h2>\n<h3>What minimum telemetry does Struct require to produce a useful investigation?<\/h3>\n<p>Struct performs best when a team already uses structured logging with consistent trace or correlation IDs, an observability platform such as Datadog, AWS CloudWatch, or GCP Logs, and an exception tracker like Sentry. If those three data sources are connected, Struct can correlate errors, map a timeline, and identify the root cause for most production incidents. Teams with sparse or unstructured logs receive partial investigations. Struct surfaces available data and flags where coverage gaps prevent a complete analysis, which becomes actionable signal for improving logging hygiene.<\/p>\n<h3>Is Struct compliant with SOC 2 and HIPAA requirements?<\/h3>\n<p>Yes. Struct is fully SOC 2 and HIPAA compliant. Log data is accessed and processed ephemerally, and Struct does not store it beyond the investigation window. For Seed-to-Series C companies operating under standard enterprise compliance requirements, this coverage is sufficient. One limitation remains: organizations with strict on-premise or zero-egress VPC policies that block any log data from leaving their internal network are not a fit for Struct\u2019s current architecture, which requires API-level access to connected observability platforms.<\/p>\n<h3>Can junior engineers run the 7 QC workflow without senior support?<\/h3>\n<p>The 7 QC sequence is designed to be repeatable regardless of seniority. Each tool has a defined input, a defined output, and a clear rule for when to apply it. The Skip vs. Use table above gives junior engineers an explicit decision framework. The main challenge for less experienced engineers sits in the data-assembly phase. Knowing which log queries to write, which trace IDs to follow, and which recent deploys matter requires context that new hires often lack. Struct addresses this by automating the data-assembly phase and encoding team-specific runbooks, so junior engineers start every alert with a fully contextualized investigation.<\/p>\n<h3>What happens when Struct investigates an incident with incomplete or malformed logs?<\/h3>\n<p>Struct surfaces the investigation based on available data and clearly flags where data gaps exist. If a service is missing trace IDs, emits unstructured log lines, or is not connected to a supported observability integration, Struct notes those gaps in the investigation output instead of fabricating inferences. This behavior makes incomplete logging visible as an operational risk. Teams often discover logging coverage gaps through Struct investigations that would otherwise stay hidden until a major incident exposed them.<\/p>\n<h2>Conclusion: Turn 3 AM Alerts into 30-Minute RCAs<\/h2>\n<p>The 7 QC tools, namely check sheet, Pareto chart, fishbone diagram, histogram, scatter diagram, flowchart, and control chart, form a complete, sequenced RCA workflow that fits production incidents. Applied in order, they take a noisy alert and produce a confirmed root cause with documented evidence in under 30 minutes. The Skip vs. Use table keeps the workflow lean by limiting you to the tools that match the incident\u2019s data profile.<\/p>\n<p>The main constraint is not the analysis sequence. It is the 20\u201330 minutes of manual data assembly that comes first. <a href=\"https:\/\/ibm.com\/think\/topics\/fmea\" target=\"_blank\" rel=\"noindex nofollow\">Structured reliability practices reduce mean time to repair<\/a> only when engineers can reach the analysis phase without spending half an hour hunting for data. Struct automates that first pass and delivers a pre-assembled, evidence-backed investigation as soon as an alert fires.<\/p>\n<p><a href=\"https:\/\/cal.com\/deepanm\/struct-demo\" target=\"_blank\">Start automating your on-call workflow today<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Learn how to apply the 7 QC tools to diagnose production incidents fast. Struct automates data assembly so your team focuses on analysis. Start free.<\/p>\n","protected":false},"author":73,"featured_media":635,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-636","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/posts\/636","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/comments?post=636"}],"version-history":[{"count":0,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/posts\/636\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/media\/635"}],"wp:attachment":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/media?parent=636"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/categories?post=636"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/tags?post=636"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}