How to Do Dynatrace Distributed Trace Analysis Effectively

Written by: Nimesh Chakravarthi, Co-founder & CTO, Struct

Key Takeaways

• Dynatrace distributed tracing gives full request visibility but often requires a manual six-step workflow that can take 45 minutes during incidents.
• Effective analysis scopes time windows, applies DQL filters, sorts by p99 latency, compares against baselines, and joins traces to logs.
• Common pitfalls include missing trace context propagation, over-filtering that hides root causes, and misinterpreting async span durations.
• Teams that adopt comprehensive observability practices reduce mean time to detection and resolution by correlating metrics, logs, and traces.
• Struct automates the entire trace analysis workflow by ingesting Dynatrace data and surfacing root causes instantly — see how Struct automates trace analysis.

Step 1: Configure Dynatrace Tracing for the Incident Window

Goal: Establish a scoped, time-bounded investigation context.
Inputs: Dynatrace environment URL, incident time window, affected service name.
Outputs: A filtered trace list scoped to the relevant service and timeframe.

The Distributed Tracing App enables analysis of trace data at both request and span levels to isolate root causes in complex microservice paths. On launch, set the time selector to bracket the incident window, typically five minutes before the alert fired to five minutes after the first symptom resolved. This boundary keeps your focus on traces that overlap with the incident instead of unrelated background traffic. Within that window, apply a service filter to narrow the scope to the affected service, because leaving the scope open to the full environment returns the maximum 1,000 records, most of which are irrelevant noise.

Practical consideration: Dynatrace data-retention windows vary by plan tier. Traces older than the retention boundary will not appear. Confirm the retention policy for the affected environment before assuming data is missing.

Step 2: Use Targeted Filters to Shrink Dynatrace Trace Results

Goal: Reduce the visible trace population to only the requests relevant to the incident.
Inputs: Hostname, Kubernetes namespace, custom span attributes, endpoint pattern.
Outputs: A result set small enough to scan visually or sort meaningfully.

The Distributed Tracing App supports DQL-based filters entered directly in the filter field, combining conditions like namespace, endpoint, and response time thresholds. Use DQL to express the exact slice of traffic you care about, such as slow requests for a specific endpoint in a single namespace. The following DQL snippet filters to slow cart-service requests in the production namespace and isolates requests exceeding five seconds in the cart flow, which is a common pattern for surfacing checkout bottlenecks:

fetch spans | filter k8s.namespace.name == "prod" AND span.name == "/cart/*" AND duration >= 5000000000 | sort duration desc

Facets in the Distributed Tracing App map to detected span attribute key-value pairs and serve as quick filters for common attributes. Start with facets for rapid iteration and quick narrowing of the dataset. Switch to raw DQL when facet cardinality becomes too high to navigate or when you need more precise combinations of attributes.

Practical consideration: High-cardinality attributes like user ID or session ID can produce filter results so narrow that the true culprit span is excluded. Filter on service-level attributes first, confirm the anomaly exists, then drill down.

Step 3: Rank Endpoints by p99 Latency and Spot Outlier Spans

Goal: Surface the slowest requests in the filtered set and identify which span is responsible.
Inputs: Filtered trace list from Step 2.
Outputs: A ranked list of traces with the highest-latency outliers at the top.

The single trace perspective provides a waterfall view of spans with duration bars, service coloring, span kind icons, and direct access to correlated logs. After sorting, open the top three to five traces and compare their waterfall views. The span that appears disproportionately wide relative to its siblings is the bottleneck candidate and deserves deeper inspection.

To identify which specific endpoints within a service are consistently slow, aggregate by endpoint name and calculate p99 latency. The following query surfaces the ten slowest endpoints in the payment service, ranked by their 99th percentile duration, so you can focus on the endpoints where 99 percent of requests complete faster and treat the top results as your highest-impact improvement targets:

fetch spans | filter service.name == "payment-service" | summarize p99_duration = percentile(duration, 99), by: {span.name} | sort p99_duration desc | limit 10

Practical consideration: Async spans such as message queue consumers and background jobs appear in the waterfall, but their wall-clock duration is not directly comparable to synchronous HTTP spans. Flag async spans before drawing latency conclusions so you do not confuse queue wait time with service performance.

Step 4: Compare Incident Traces Against a Stable Baseline

Goal: Detect regressions by contrasting the incident window against a stable reference period.
Inputs: Incident time window, a known-good baseline window (same service, same day of week, prior week).
Outputs: A side-by-side delta showing which spans degraded and by how much.

The Dynatrace Compare feature allows selection of two time windows for the same filtered query. Set the baseline to the equivalent period seven days prior so traffic patterns match as closely as possible. Any span whose p99 latency increased by more than 20 percent between windows is a regression candidate. Comprehensive observability implementations reduce mean time to detection from hours to minutes by providing unified correlation across metrics, logs, and traces instead of manual investigation across disconnected systems. The Compare feature applies that principle at the trace layer by highlighting which spans changed behavior.

Practical consideration: Traffic volume differences between the baseline and incident windows can inflate latency deltas artificially. Normalize by request count before treating a delta as a confirmed regression.

Step 5: Join Dynatrace Traces to Logs with DQL

Goal: Correlate the outlier trace to its log entries to identify the specific error or state change that caused the latency spike.
Inputs: Trace ID from the outlier span, log data in Grail.
Outputs: A unified timeline of span events and log lines for the same request.

A practical best practice is to ensure trace IDs appear in logs so an engineer can pivot from a log entry to the full trace during incident response. Correlation IDs such as trace_id and span_id included in structured logs connect logs, traces, and metrics into a unified view of the same event, enabling teams to move from anomaly detection to root-cause diagnosis. This correlation turns a slow span into a concrete error message or state transition.

The following DQL query fetches all log lines for a specific trace ID so you can read the exact sequence of events that occurred during the problematic request:

fetch logs | filter trace_id == "4bf92f3577b34da6a3ce929d0e0e4736" | sort timestamp asc | fields timestamp, severity, content, service.name

After you confirm which logs align with the slow traces, extend the query to answer impact questions. The next example joins slow checkout spans with error logs and summarizes impact by loyalty tier, which helps you quantify business risk and prioritize fixes:

fetch spans | filter service.name == "checkout-service" AND duration >= 3000000000 | lookup [fetch logs | filter log.level == "ERROR"], sourceField: trace_id, lookupField: trace_id | summarize error_count = count(), by: {user.loyalty_status}

Practical consideration: Using structured logging in JSON format and consistent log levels, DEBUG for development and INFO or WARN or ERROR in production, enables precise queries, automatic correlation via trace_id, and faster incident triage. If logs are unstructured, the join will fail silently and return empty results, which hides the real cause.

Try Struct’s automated investigation workflow

Step 6: Send Dynatrace Evidence to Struct for Automated Root Cause

Goal: Hand off the correlated trace-and-log evidence to an automated investigation platform that produces a confirmed root cause without additional manual steps.
Inputs: Trace ID, DQL query results, incident time window, Dynatrace environment connection.
Outputs: A dynamically generated Struct dashboard containing impact summary, root cause, timeline, and suggested fix.

This final step removes the manual correlation work from the earlier steps by handing the entire investigation to an automated platform. Struct customers working at large scale with many services report an 80% reduction in triage time. Setup takes ten minutes: authenticate Slack or PagerDuty as the alert source, connect GitHub for code context, and link Dynatrace or the relevant observability platform. From that point, Struct intercepts every alert automatically.

Deepan Mehta, co-founder of Struct, stated: “Struct gets you from alert → root cause before you even open your laptop.” Struct is SOC 2 and HIPAA compliant, which makes it suitable for fintech and healthcare engineering teams with strict data-handling requirements. Its Slack-native conversational AI allows engineers to ask follow-up questions such as “pull logs from five minutes prior” or “verify if this impacts user segment X” directly in the incident thread, so they stay in one place while refining the investigation.

Practical consideration: Struct processes logs ephemerally. Data is not retained beyond the investigation session, which satisfies most Seed-to-Series-C compliance requirements without requiring on-premise deployment.

Integrate Dynatrace and Struct with PagerDuty, Slack, and GitHub

When a PagerDuty alert fires, Struct begins its investigation immediately in the background. By the time the on-call engineer acknowledges the page, the investigation described earlier is already complete and waiting in Slack. Distributed tracing becomes more effective when tied into existing observability workflows such as alerts, SLOs, and incident response, rather than treated as a standalone tool. Once Struct confirms root cause, it can hand off context to a coding agent or generate a pull request directly against the affected GitHub repository, which closes the loop from alert to code fix.

Measure MTTR Gains and Run Weekly Trace Reviews

Companies implementing comprehensive observability solutions often see significant reductions in mean time to resolution. The MTTR improvements mentioned earlier apply directly to this workflow, so track your team’s progress weekly using Dynatrace’s built-in problem analytics alongside Struct’s investigation logs. A weekly thirty-minute review of recurring trace patterns, such as services that appear in the p99 outlier list more than twice in a week, surfaces systemic issues before they become incidents. Use the DQL grouping function to aggregate by service and endpoint across the prior seven days and prioritize the top three offenders for proactive remediation.

Common Pitfalls to Avoid in Dynatrace Trace Analysis

Missing trace context propagation. OpenTelemetry support is recommended to avoid vendor lock-in while correlating traces with logs, metrics, and deployments. Services that do not propagate W3C TraceContext headers break the trace chain, which makes the waterfall view appear to end prematurely. Audit every service boundary for header propagation before relying on waterfall analysis.

Over-filtering that hides the true culprit. Applying too many AND conditions in DQL can exclude the exact span responsible for the incident. Start broad, confirm the anomaly exists in the result set, then narrow incrementally.

Ignoring async versus sync call patterns. Traces are best for locating latency, dependencies, and cascading failures, but async consumers introduce artificial latency in the waterfall that reflects queue depth, not service performance. Label async spans explicitly in instrumentation to avoid misattributing queue wait time as service latency.

Frequently Asked Questions

What minimum Dynatrace feature flags are required to run this workflow?

The Distributed Tracing App requires Dynatrace OneAgent or OpenTelemetry instrumentation deployed on all services in the call chain, with Grail data ingestion enabled. DQL queries require the Grail storage tier to be active on the environment. The Compare feature is available on Dynatrace SaaS environments running the current-generation platform. Ensure that span attribute ingestion is not throttled at the environment level, because attribute limits prevent custom facets from appearing in the filter panel.

What are the data egress considerations when using Struct alongside Dynatrace?

Struct accesses logs, traces, and metrics via authenticated API connections to your observability platforms. Data is processed ephemerally during the investigation session and is not stored on Struct’s infrastructure beyond that window. Struct is SOC 2 and HIPAA compliant, which satisfies most cloud security requirements. For organizations with strict VPC-boundary requirements that prohibit any log data from leaving internal infrastructure, Struct currently requires external API access to function and would not be the right fit until an on-premise deployment option is available.

How can junior engineers safely consume Struct-generated dashboards without deep system knowledge?

Struct’s dynamically generated dashboards present a structured narrative: blast radius first, then a unified timeline, then the identified root cause, then suggested fixes. Junior engineers do not need to interpret raw DQL output or cross-reference five tools. They review the Struct dashboard in Slack, confirm the impact scope, and either apply the suggested fix or escalate with full context already assembled. Teams can also encode their internal runbooks directly into Struct, so the AI follows the same investigation steps a senior engineer would, which gives newer team members a reliable, repeatable starting point for every alert type.

Conclusion: Shift from Manual Trace Hunting to Automated Root Cause

The six-step workflow, configure the app, filter strategically, sort by p99, compare against a baseline, join traces to logs with DQL, and hand off to Struct, converts a 45-minute manual investigation into a process that completes in under ten minutes. A mature observability practice follows the SRE investigation flow: metrics detect the anomaly, traces locate the bottleneck, and logs diagnose the cause. Struct automates that entire flow the moment an alert fires, so the root cause is waiting in Slack before the engineer reaches for their laptop.

Stop burning engineering hours on manual trace hunting at 3 AM. Automate your on-call runbook and let Struct handle the next investigation.