Written by: Nimesh Chakravarthi, Co-founder & CTO, Struct | Last updated: August 24, 2026
Key Takeaways
- Most CloudWatch setups page on resource metrics instead of user-visible symptoms, which creates unnecessary 3 AM wake-ups for non-issues.
- Design symptom-first alarms around the four golden signals, use composite logic to suppress noise, and apply SLO burn-rate math for precise alerting.
- Split SNS topics by severity, route critical pages through PagerDuty, and test every notification path end to end on a quarterly schedule.
- Incident resolution verification confirms that user impact has ended, not just that an alarm returned to the OK state.
- Struct automates the investigation layer on top of CloudWatch so engineers receive root-cause summaries instead of raw alerts. Start an automated on-call trial.
The Problem: CloudWatch Paging on Causes Instead of Symptoms
Incident responders often receive multiple alerts per shift, which creates significant alert fatigue in many enterprises. The root cause is alarm design, not alert volume. Engineers alarm on causes such as CPU, memory, and queue depth because those metrics are easy to instrument. A single deployment failure that degrades CPU, error rate, p99 latency, and queue depth can trigger four separate pages, creating alert fatigue from a single event.
On-call stress can contribute to burnout and attrition on engineering teams. The AWS Well-Architected Framework calls out excessive non-critical alerts and resulting alert fatigue as a common anti-pattern.
The Solution: Paging Only on User-Visible Symptoms
Symptom alarms fire on user-experienced conditions such as failed requests, excessive latency, stalled pipelines, and missed job deadlines. Reserving paging exclusively for symptom-based alarms reduces alert fatigue because a single symptom alarm can cover many underlying causes. Cause alarms for disk fill, throttled tables, and connection counts route to tickets or dashboards for post-page inspection, never to PagerDuty.
Implementing this symptom-first approach in CloudWatch requires configuring alarms differently based on whether they represent user impact or internal resource state.
Building Symptom and Cause Alarms in CloudWatch
Define symptom alarms on the four golden signals (latency, traffic, errors, saturation) using Metric Math expressions or Log Insights queries scoped to user-visible endpoints. For the same underlying infrastructure, configure cause alarms that monitor resource state, and wire those alarms to a ticket-filing SNS topic with no paging action. Before adding any new alarm to either category, confirm in one sentence the immediate action to take if it fires. If the answer is “just watch it,” the metric belongs on a dashboard, not in your alarm inventory.
| Dimension | Symptom Alarm | Cause Alarm | Routing |
|---|---|---|---|
| Example metric | ALB HTTPCode_Target_5XX rate, p99 latency | EC2 CPUUtilization, RDS queue depth | – |
| User impact | Direct, errors or slowness visible to users | Indirect, internal resource state only | – |
| SNS destination | PagerDuty or on-call paging topic | Ticket-filing topic or dashboard only | Split by severity |
| Evaluation config | 3-of-5 DatapointsToAlarm, TreatMissingData = breaching for health checks | 5-of-5 or dashboard only, TreatMissingData = notBreaching | M-of-N required |
Alt text for architecture diagram: A flow diagram showing metric alarms for ALB 5xx rate, p99 latency, and queue age feeding into a CloudWatch composite alarm with AND logic, which routes to an SNS paging topic connected to PagerDuty. Separate cause alarms for CPU and disk feed a second SNS topic connected to a ticketing system. A maintenance-window suppressor alarm blocks the composite alarm during planned downtime.
Configuring CloudWatch SLO Burn Rate Alarms
Use CloudWatch Metric Math to compute burn rate as (1 - success_rate) / error_budget_fraction from request-success data. For a 99.9% SLO the monthly error budget is roughly 43 minutes, and spending that budget faster than 1/30 of a day indicates the SLO is at risk. Create two child alarms, a fast-burn alarm (14.4× over 5-minute and 1-hour windows) and a slow-burn alarm (6× over 30-minute and 6-hour windows). Combine them with AND logic in a composite alarm so both windows must breach before a page fires.
CloudWatch Application Signals supplies the SLO layer automatically on top of discovered services. It provides standardized latency and availability SLIs, error-budget tracking, and the attainment metrics that burn-rate alarms evaluate against. For services not yet on Application Signals, define the SLI as a Metric Math expression over existing CloudWatch metrics.
Reducing CloudWatch Alert Fatigue with Composite Alarms
Route all child metric alarms without direct SNS actions, and send pages exclusively through composite alarms that require multiple symptom conditions to be true simultaneously. An example composite expression pages only when CPUUtilization-High, HTTPCode_Target_5XX-Elevated, and TargetResponseTime-P99-Elevated are all in ALARM state simultaneously. Add a suppressor alarm for maintenance windows using the NOT ALARM("maintenance-window-active") clause to prevent planned changes from generating pages.
If the alert-to-actionable-incident conversion rate sits below 20%, there is a noise problem, and the target operating range is 30–50% actionable. Composite alarms are the primary CloudWatch mechanism for reaching that range without suppressing genuine incidents.
CloudWatch to PagerDuty Integration Best Practices
Route only composite symptom alarms to PagerDuty. Create a dedicated SNS topic for critical pages, subscribe the PagerDuty HTTPS endpoint to that topic, and confirm the subscription before treating the path as live. Validate that the TopicArn exactly matches the route configuration, the SigningCertURL uses HTTPS and belongs to Amazon SNS, and the RSA signature matches the canonical signing string.
Configure both ALARM and OK notifications on every alarm so PagerDuty can auto-resolve incidents. CloudWatch alarms fail to resolve in the incident platform if the ALARM and OK messages contain different AlarmArn values. Both states must share the identical ARN for deduplication. Add the PagerDuty Lambda or webhook as an additional SNS subscriber instead of replacing existing subscriptions so current email or chat paths remain intact.
Testing CloudWatch Alarm Notification Paths
Run the following command against a purpose-built test alarm during a scheduled maintenance window:
aws cloudwatch set-alarm-state \ --alarm-name prod-alb-5xx-rate \ --state-value ALARM \ --state-reason "quarterly drill"An incident does not care that a CloudWatch alarm changed state if no human or system actually received a notification, because the failure occurs at the boundary between signal and delivery. Verify each hop: the alarm history shows an action attempt, the SNS topic metrics show a publish, and the notification lands in the on-call channel within seconds.
Check the full notification content, not just delivery. Verify the message subject, account markers, alarm name, reason, and runbook or dashboard links so the notification is usable by an on-call responder without additional context hunting. Run these drills quarterly and log acknowledgment time as a metric.
Verifying Incident Resolution, Not Just Alarm State
Alarm state returning to OK does not guarantee that the incident is resolved. A CloudWatch alarm transitions to OK when the metric drops below threshold, but the underlying cause may persist, a different signal may still be degraded, or the fix may not have propagated across all instances. Incident resolution verification requires checking observability data, not just alarm state, to confirm that user impact has actually ended.
Struct’s Incident Tracker runs an approximately 1-minute automated verification loop against live observability data to confirm an incident is actually resolved before closing it. This loop closes the gap between “alarm cleared” and “incident verified closed.” That distinction matters for SLA compliance and post-incident accuracy. Incident resolution verification is the category Struct is purpose-built to own, and no other tool currently provides a dedicated automated verification loop at this granularity.
How Struct Adds an Automated Investigation Layer
CloudWatch surfaces the signal, and Struct performs the investigation. When a composite alarm fires and PagerDuty pages an engineer, Struct has already queried CloudWatch logs, correlated trace IDs, mapped the blast radius, and generated a root-cause summary before the engineer opens their laptop. Struct sits on top of CloudWatch as an investigation layer and does not replace it.
Arcana cut investigation time from 30 minutes to 2 minutes using Struct, reclaims 56 engineer-hours per month, and runs 2,100+ automated investigations monthly. That outcome comes from pairing a well-designed CloudWatch alarm pipeline with an automated first-pass investigation layer. Without the CloudWatch pipeline, Struct receives noisy signals. Without Struct, engineers still triage manually after the right alarm fires.
Setup takes under 10 minutes. Authenticate Slack or PagerDuty as the trigger source, connect CloudWatch as the observability context, and connect GitHub for code correlation. Struct then intercepts every alarm that reaches your on-call channel, runs the investigation automatically, and delivers a dynamically generated dashboard with impact scope, root cause, and suggested fixes. Struct is SOC 2 Type II and HIPAA compliant.
Set up automated incident investigation and let AI handle your next on-call triage.
Frequently Asked Questions
What is the difference between a symptom alarm and a cause alarm in CloudWatch?
A symptom alarm fires on a condition that directly affects users, such as elevated error rates, p99 latency breaches, failed health checks, or missed job deadlines. A cause alarm fires on an internal resource state such as CPU utilization, memory pressure, or queue depth. Symptom alarms belong on paging paths because they confirm user impact. Cause alarms belong on dashboards or ticket-filing paths because high CPU alone does not mean users are affected. The practical test is simple: if you cannot write a one-sentence immediate action for the alarm, it is a cause alarm and should not page.
How many CloudWatch alarms should a production SaaS service have?
A well-designed production service needs roughly a dozen paging alarms focused on customer-visible symptoms such as ALB 5xx error rate, target p95 or p99 latency, unhealthy host count, queue age, replica lag, backup job failures, certificate expiry, and disk fill rate. Additional cause alarms can exist for dashboards and tickets, but the paging set should stay small. The goal is the 30–50% actionable rate discussed earlier. If fewer than 30% of pages require immediate action, the alarm set needs pruning. Composite alarms are the primary mechanism for keeping the paging set small without missing genuine incidents.
How do you prevent CloudWatch alarms from firing during deployments?
Use CloudWatch alarm mute rules to suppress actions during a defined deployment window. Mute rules temporarily block alarm actions while the alarm continues to evaluate state, so the alarm history remains accurate and the suppression stays time-bounded. As an alternative, add a suppressor alarm to your composite alarm rule using NOT ALARM(“maintenance-window-active”). Set that suppressor alarm to ALARM state through the CLI at the start of a deployment window, then reset it to OK when the deployment completes. Both approaches prevent expected metric changes from generating pages without disabling the alarms entirely.
What does Struct do that CloudWatch cannot?
CloudWatch detects and signals anomalies, and Struct investigates them. When a CloudWatch composite alarm fires, Struct automatically queries logs, correlates trace IDs across your stack, maps the blast radius, identifies the root cause, and delivers a structured dashboard before an engineer manually opens a single tool. CloudWatch cannot correlate across GitHub code changes, Sentry exceptions, and log data simultaneously, and it cannot verify that a fix actually resolved the incident against live observability data. Struct’s Incident Tracker adds a continuous automated verification loop that confirms resolution against real data rather than relying solely on alarm state transitions.
Is Struct secure enough for fintech workloads with strict compliance requirements?
Struct is SOC 2 Type II and HIPAA compliant, which covers the compliance requirements of the vast majority of Series A–C fintech companies. Logs and context are accessed and processed ephemerally, and they are not stored beyond the investigation window. Full compliance documentation is available at trust.struct.ai. If your organization requires full on-premise deployment with zero data leaving your VPC, Struct’s current architecture requires external integration access and would not be the right fit until on-prem support becomes available on the Enterprise tier.
Conclusion
A reliable CloudWatch on-call pipeline has seven components: pruned alarm inventory, golden-signal symptom alarms, M-of-N evaluation with deliberate TreatMissingData settings, composite alarms with suppression logic, SLO burn-rate alarms on dual windows, severity-split SNS routing to PagerDuty, and quarterly end-to-end notification path tests. Each step reduces the ratio of noise to actionable pages. The final step, incident resolution verification, closes the loop by confirming that user impact has actually ended, not just that a metric crossed back below a threshold.
Struct sits at the end of that pipeline as the automated investigation layer. When the right alarm fires, Struct delivers root cause and blast radius before your engineer reaches for their laptop. The Arcana results mentioned earlier show the compound effect of pairing well-designed alarms with automated investigation.
Stop burning your best engineers on 3 AM log-hunting expeditions. Reduce the time your team spends triaging issues by 80% and give them their product velocity back. Set up Struct in under 10 minutes and let AI handle your next on-call investigation. Start your free trial and connect your first integration.
