{"id":670,"date":"2026-06-23T05:00:22","date_gmt":"2026-06-23T05:00:22","guid":{"rendered":"https:\/\/struct.ai\/articles\/grafana-tempo-production-trace-analysis\/"},"modified":"2026-06-23T05:00:22","modified_gmt":"2026-06-23T05:00:22","slug":"grafana-tempo-production-trace-analysis","status":"publish","type":"post","link":"https:\/\/struct.ai\/articles\/grafana-tempo-production-trace-analysis\/","title":{"rendered":"How to Use Grafana Tempo for Production Trace Analysis"},"content":{"rendered":"<p><em>Written by: Nimesh Chakravarthi, Co-founder &amp; CTO, Struct<\/em><\/p>\n<h2 id=\"key-takeaways\">Key Takeaways for Using Tempo in Production<\/h2>\n<ul>\n<li>Grafana Tempo stores traces in object storage and uses TraceQL for fast span-level queries, which suits production incident response.<\/li>\n<li>Configure Alloy with resilient queue and retry settings so it buffers traces and prevents span loss during Tempo restarts or high load.<\/li>\n<li>Apply tail-based sampling to retain 100% of error and slow traces while cutting healthy trace storage by up to 90\u201395%.<\/li>\n<li>Enable Tempo\u2019s metrics-generator for RED metrics with exemplars and configure Loki derived fields so logs link directly to traces.<\/li>\n<li>Struct ingests correlated signals from Tempo and posts root cause analysis to Slack; <a href=\"https:\/\/cal.com\/deepanm\/struct-demo\" target=\"_blank\">see how Struct automates your incident response<\/a><\/li>\n<\/ul>\n<h2>Buffering Tempo Writes with Alloy \/ OpenTelemetry Collector<\/h2>\n<p>Running the OpenTelemetry Collector or Grafana Alloy as a sidecar or DaemonSet decouples your services from Tempo\u2019s write path. When Tempo slows down or restarts, the collector absorbs the burst instead of dropping spans at the SDK level.<\/p>\n<p>Use these queue and retry settings for Alloy\u2019s <code>otelcol.exporter.otlp<\/code> block:<\/p>\n<pre>otelcol.exporter.otlp \"tempo\" { client { endpoint = \"tempo:4317\" tls { insecure = true } } sending_queue { enabled = true num_consumers = 8 queue_size = 10000 } retry_on_failure { enabled = true initial_interval = \"5s\" max_interval = \"30s\" max_elapsed_time = \"300s\" } } <\/pre>\n<p>Set <code>queue_size<\/code> to at least 10,000 spans for services that emit more than 5,000 spans per second, so the buffer absorbs burst traffic when Tempo is slow. Increase <code>num_consumers<\/code> to match available CPU cores on the collector pod, which lets the collector drain that queue in parallel. This buffering matters for Struct because it reads traces directly from Tempo\u2019s HTTP API, and dropped spans in the collector never reach Tempo for correlation against a firing alert.<\/p>\n<h2>Tail-Based Sampling for Error-Focused Retention<\/h2>\n<p>With the collector properly buffered to prevent span loss, you can apply tail-based sampling confidently. Head-based sampling makes decisions before a trace completes, so errors and slow traces are discarded at the same rate as healthy ones. Tail-based sampling waits for the full trace, then applies policy rules that keep 100% of error traces and a configurable percentage of healthy ones.<\/p>\n<p>A practical policy for a service emitting 50,000 traces per minute at $0.30 per GB stored:<\/p>\n<table>\n<thead>\n<tr>\n<th>Policy<\/th>\n<th>Condition<\/th>\n<th>Keep Rate<\/th>\n<th>Estimated Storage Impact<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Error traces<\/td>\n<td><code>status = error<\/code><\/td>\n<td>100%<\/td>\n<td>Baseline (errors are typically &lt;5% of volume)<\/td>\n<\/tr>\n<tr>\n<td>Slow traces<\/td>\n<td><code>duration &gt; 2s<\/code><\/td>\n<td>100%<\/td>\n<td>+5\u201310% of total volume<\/td>\n<\/tr>\n<tr>\n<td>Healthy traces<\/td>\n<td>All others<\/td>\n<td>5\u201310%<\/td>\n<td>Reduces healthy-trace storage by 90\u201395%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>At 5% effective sampling on healthy traces, a team storing 500 GB per month of raw traces can reduce that to roughly 50\u201375 GB per month after accounting for 100% retention of errors and slow spans. Configure this in the <code>otelcol.processor.tail_sampling<\/code> block in Alloy, and set <code>decision_wait: 30s<\/code> so late-arriving spans complete before the policy evaluates. Struct\u2019s investigation engine benefits directly because 100% of error traces remain available, so Struct does not hit gaps when correlating a Sentry exception to a Tempo trace ID.<\/p>\n<h2>Using Metrics-Generator for RED Metrics and Exemplars<\/h2>\n<p>Tempo\u2019s built-in metrics-generator derives Rate, Error, and Duration (RED) metrics from ingested spans without a separate instrumentation pass. Enable it in <code>tempo.yaml<\/code>:<\/p>\n<pre>metrics_generator: registry: external_labels: source: tempo storage: path: \/var\/tempo\/generator\/wal remote_write: - url: http:\/\/prometheus:9090\/api\/v1\/write processor: service_graphs: dimensions: [http.method, http.status_code] span_metrics: dimensions: [http.method, http.status_code, http.target] enable_target_info: true <\/pre>\n<p>The <code>service_graphs<\/code> processor emits a directed graph of service-to-service call rates and error rates as Prometheus metrics. The <code>span_metrics<\/code> processor emits per-operation histograms. Both processors attach exemplars, which are trace IDs embedded in Prometheus metric samples, so a Grafana panel that shows a p95 latency spike links directly to a representative trace with one click. Struct ingests these exemplar-linked metrics from Prometheus and uses the embedded trace IDs to pull the exact Tempo trace that caused the anomaly, which removes the manual \u201cfind a trace from that time window\u201d step.<\/p>\n<h2>Linking Loki Logs to Tempo Traces<\/h2>\n<p>Loki derived fields create clickable trace ID links inside log lines. Configure them in the Loki datasource in Grafana:<\/p>\n<pre>derivedFields: - name: TraceID matcherRegex: \"trace_id=(\\w+)\" url: \"$${__value.raw}\" datasourceUid: tempo-uid urlDisplayLabel: \"Open in Tempo\" <\/pre>\n<p>Any log line that contains <code>trace_id=&lt;id&gt;<\/code> renders a button that opens the full trace in Tempo. For structured JSON logs, use <code>matcherRegex: '&quot;traceId&quot;:&quot;(\\w+)&quot;'<\/code>. The investigation phase, which includes identifying the component causing an outage, consumes 60\u201380% of MTTR in distributed systems, so removing manual copy and paste between Loki and Tempo directly attacks that majority. Struct reads both Loki logs and Tempo traces and uses the same trace ID linkage to build a unified incident timeline automatically.<\/p>\n<h2>TraceQL Patterns for Errors and Slow Spans<\/h2>\n<p>TraceQL is Tempo\u2019s purpose-built query language for span-level filtering across millions of traces. TraceQL supports filtering by service name, span duration, status codes, HTTP methods, and custom attributes, and it returns results incrementally through streaming search instead of waiting for a full dataset scan.<\/p>\n<p>Use these TraceQL patterns during production incidents:<\/p>\n<table>\n<thead>\n<tr>\n<th>Scenario<\/th>\n<th>TraceQL Query<\/th>\n<th>When to Use<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>All error traces<\/td>\n<td><code>{ status = error }<\/code><\/td>\n<td>First query on any alert<\/td>\n<\/tr>\n<tr>\n<td>Slow checkout spans<\/td>\n<td><code>{ .service.name = \"checkout\" &amp;&amp; duration &gt; 2s }<\/code><\/td>\n<td>p95 latency alert on checkout<\/td>\n<\/tr>\n<tr>\n<td>Auth-to-payment failures<\/td>\n<td><code>{ .service.name = \"auth\" } &amp;&amp; { .service.name = \"payment\" }<\/code><\/td>\n<td>Cross-service dependency failures<\/td>\n<\/tr>\n<tr>\n<td>HTTP 5xx on specific route<\/td>\n<td><code>{ .http.status_code &gt;= 500 &amp;&amp; .http.target = \"\/api\/order\" }<\/code><\/td>\n<td>Route-level error spike<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>TraceQL allows multiple conditions in a single query, which narrows traces across large datasets during failure investigations in distributed systems. Struct encodes these patterns into its runbook engine, so when a checkout latency alert fires, Struct runs the relevant TraceQL query and surfaces the slowest spans in its generated dashboard without manual input.<\/p>\n<h2>Connecting Tempo Traces to Pyroscope Profiles<\/h2>\n<p>Grafana Pyroscope continuous profiling integrates with Tempo through the <code>profileTypes<\/code> link in the trace view. Enable this by setting a matching <code>profileURL<\/code> in the Tempo datasource and ensuring your services emit profiles with the same <code>service.name<\/code> attribute used in spans. When a slow span appears in Tempo, a \u201cProfiles\u201d button opens the CPU or memory flame graph for that service during the trace\u2019s time window.<\/p>\n<p>This workflow narrows root cause from \u201cthe checkout service was slow\u201d to \u201ca specific function consumed 94% of CPU during that request.\u201d Struct uses the Pyroscope API to attach profile snapshots to its incident report when a trace shows anomalous duration, which gives engineers a code-level answer without switching tools.<\/p>\n<h2>Tempo Production Readiness Checklist<\/h2>\n<table>\n<thead>\n<tr>\n<th>Area<\/th>\n<th>Setting<\/th>\n<th>Recommended Value<\/th>\n<th>Status<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Collector<\/td>\n<td>Queue size<\/td>\n<td>\u2265 10,000<\/td>\n<td>\u2610<\/td>\n<\/tr>\n<tr>\n<td>Collector<\/td>\n<td>Retry max elapsed time<\/td>\n<td>300s<\/td>\n<td>\u2610<\/td>\n<\/tr>\n<tr>\n<td>Sampling<\/td>\n<td>Error trace retention<\/td>\n<td>100%<\/td>\n<td>\u2610<\/td>\n<\/tr>\n<tr>\n<td>Sampling<\/td>\n<td>Healthy trace retention<\/td>\n<td>5\u201310%<\/td>\n<td>\u2610<\/td>\n<\/tr>\n<tr>\n<td>Metrics-generator<\/td>\n<td>Service graphs enabled<\/td>\n<td>true<\/td>\n<td>\u2610<\/td>\n<\/tr>\n<tr>\n<td>Metrics-generator<\/td>\n<td>Exemplars enabled<\/td>\n<td>true<\/td>\n<td>\u2610<\/td>\n<\/tr>\n<tr>\n<td>Loki<\/td>\n<td>Derived fields regex<\/td>\n<td>trace_id pattern set<\/td>\n<td>\u2610<\/td>\n<\/tr>\n<tr>\n<td>Pyroscope<\/td>\n<td>profileURL linked<\/td>\n<td>Matching service.name<\/td>\n<td>\u2610<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Common Tempo and Struct Issues<\/h2>\n<table>\n<thead>\n<tr>\n<th>Symptom<\/th>\n<th>Likely Cause<\/th>\n<th>Fix<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>\u201cNo traces found\u201d for a known error<\/td>\n<td>Head-based sampling dropped the trace<\/td>\n<td>Switch to tail-based sampling with 100% error retention<\/td>\n<\/tr>\n<tr>\n<td>Exemplar links return 404 in Grafana<\/td>\n<td>Trace ID in exemplar not ingested by Tempo<\/td>\n<td>Verify collector queue is not dropping spans, then check Tempo ingester logs<\/td>\n<\/tr>\n<tr>\n<td>TraceQL query times out<\/td>\n<td>Querying too large a time window<\/td>\n<td>Narrow the window to 1 hour or less and use streaming search<\/td>\n<\/tr>\n<tr>\n<td>Derived field link does not appear in Loki<\/td>\n<td>Regex does not match log format<\/td>\n<td>Test the regex against a raw log line in Grafana Explore<\/td>\n<\/tr>\n<tr>\n<td>Metrics-generator metrics missing<\/td>\n<td>Remote write endpoint unreachable<\/td>\n<td>Check the Prometheus remote write URL and network policy<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Tempo vs. Jaeger in Production Environments<\/h2>\n<p>The configurations above prepare Tempo for production incident response and automated investigations. Teams that evaluate tracing backends often compare Tempo with Jaeger, which is the other major open-source option. <a href=\"https:\/\/www.jaegertracing.io\/docs\/2.dev\/storage\/\" target=\"_blank\" rel=\"noindex nofollow\">Jaeger\u2019s native storage primarily uses Cassandra, Elasticsearch, or OpenSearch as distributed backends<\/a>, which require cluster management and tend to scale vertically. Tempo writes directly to object storage such as S3, GCS, or Azure Blob, which separates compute from storage and reduces operational overhead.<\/p>\n<p>For tail-based sampling, Jaeger\u2019s collector supports it through the <code>jaeger-agent<\/code> pipeline, but the configuration is less composable than Alloy\u2019s processor chain. Tempo\u2019s metrics-generator also has no direct Jaeger equivalent, so RED metrics from Jaeger require a separate Prometheus instrumentation layer.<\/p>\n<table>\n<thead>\n<tr>\n<th>Dimension<\/th>\n<th>Grafana Tempo<\/th>\n<th>Jaeger<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Storage backend<\/td>\n<td>Object storage (S3\/GCS\/Azure)<\/td>\n<td>Cassandra, Elasticsearch or OpenSearch<\/td>\n<\/tr>\n<tr>\n<td>Tail-based sampling<\/td>\n<td>Native via Alloy processor<\/td>\n<td>Requires separate collector config<\/td>\n<\/tr>\n<tr>\n<td>RED metrics generation<\/td>\n<td>Built-in metrics-generator<\/td>\n<td>Requires external instrumentation<\/td>\n<\/tr>\n<tr>\n<td>Query language<\/td>\n<td>TraceQL (span-level filtering)<\/td>\n<td>Jaeger UI search (tag-based)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Tempo\u2019s native exemplar linking and metrics-generator reduce the manual correlation overhead discussed earlier more directly than Jaeger\u2019s architecture allows. <a href=\"https:\/\/traversal.com\/blog\/what-is-mttr\" target=\"_blank\" rel=\"noindex nofollow\">High-performing SRE teams typically remediate major incidents in under an hour<\/a>, and the configurations in this guide push toward that benchmark by eliminating the manual steps that consume most investigation time. <a href=\"https:\/\/cal.com\/deepanm\/struct-demo\" target=\"_blank\">See how Struct helps teams hit that sub-hour MTTR target<\/a><\/p>\n<h2>Conclusion: Turning Tempo into an Incident Signal Engine<\/h2>\n<p>A properly configured Grafana Tempo stack with Alloy queue and retry settings, tail-based sampling at 100% error retention, metrics-generator exemplars, Loki derived fields, and TraceQL incident patterns turns traces into an active incident signal. Major outages carry significant financial costs, so every minute of triage time you remove has direct financial impact.<\/p>\n<p>Struct sits at the end of this pipeline, ingesting the correlated signals Tempo produces and delivering root cause analysis to Slack before the on-call engineer finishes reading the alert. The 3 AM log-hunting expedition becomes a 5-minute review. <a href=\"https:\/\/cal.com\/deepanm\/struct-demo\" target=\"_blank\">Book a demo to see this pipeline in action<\/a><\/p>\n<h2>Frequently Asked Questions<\/h2>\n<h3>Does Struct work if our Tempo traces are incomplete or inconsistently instrumented?<\/h3>\n<p>Struct performs best when trace IDs propagate consistently across services and logs contain matching trace ID fields. If instrumentation is partial, such as only certain services emitting spans, Struct correlates what is available and flags gaps in the investigation report. The tail-based sampling configuration in this guide keeps error traces at 100% retention, which gives Struct the highest-value data even when overall instrumentation coverage is uneven.<\/p>\n<p>Teams with significant instrumentation gaps should add OpenTelemetry SDK instrumentation to their highest-traffic services first, because those services generate the most actionable traces during incidents.<\/p>\n<h3>How does Struct handle the volume of traces generated by a high-throughput production service?<\/h3>\n<p>Struct queries Tempo\u2019s HTTP API instead of ingesting raw span streams, so raw trace volume does not affect Struct directly. When an alert fires, Struct runs targeted TraceQL queries scoped to the alert\u2019s time window and affected service, which retrieves only the relevant traces.<\/p>\n<p>The tail-based sampling configuration described earlier preserves 100% of error and slow traces, which are the only traces Struct needs during an incident investigation, while dramatically reducing storage costs on healthy traffic. This approach keeps Struct\u2019s query latency low regardless of overall ingestion rate.<\/p>\n<h3>Can Struct consume metrics-generator exemplars directly from Prometheus?<\/h3>\n<p>Yes. Struct integrates with Prometheus as an observability source. When metrics-generator emits RED metrics with embedded exemplar trace IDs, Struct reads those exemplars during its investigation and uses the trace IDs to pull the corresponding Tempo traces automatically.<\/p>\n<p>This mechanism allows Struct to jump from a p95 latency spike on a Prometheus alert directly to the specific slow trace without any manual lookup. Enabling the <code>span_metrics<\/code> processor with exemplars, as shown in the configuration above, is the prerequisite for this workflow.<\/p>\n<h3>Is there a minimum Grafana Tempo version required for the configurations in this guide?<\/h3>\n<p>The metrics-generator with exemplar support requires <a href=\"https:\/\/grafana.com\/blog\/new-in-grafana-tempo-1-4-introducing-the-metrics-generator\/\" target=\"_blank\" rel=\"noindex nofollow\">Tempo 1.4 or later<\/a>. TraceQL streaming search requires <a href=\"https:\/\/grafana.com\/docs\/tempo\/latest\/release-notes\/v2-2\/?pg=blog&amp;plcmt=body-txt\" target=\"_blank\" rel=\"noindex nofollow\">Tempo 2.2 or later<\/a>. Tail-based sampling through Grafana Alloy uses the <code>otelcol.processor.tail_sampling<\/code> component. Trace-to-profiles linking with Pyroscope requires a compatible Pyroscope deployment.<\/p>\n<p>Most teams that run Tempo on a current Grafana Cloud stack or a self-hosted deployment from 2024 onward already meet these feature requirements and do not need an upgrade.<\/p>\n<h3>How long does it take to connect Struct to an existing Tempo and Grafana setup?<\/h3>\n<p>Struct\u2019s initial setup usually takes under 10 minutes. Connecting Grafana as an observability source requires a service account token with read access to Tempo and Prometheus datasources. After connection, Struct immediately begins listening to your configured Slack or PagerDuty alerting channels.<\/p>\n<p>The first automated investigation runs the next time an alert fires. No changes to your existing Tempo configuration are required for basic integration, although enabling the metrics-generator and derived fields described in this guide significantly improves the depth of Struct\u2019s automated root cause analysis.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Master Grafana Tempo for production tracing with TraceQL, tail-based sampling &amp; RED metrics. Struct automates root cause analysis. Book a demo today.<\/p>\n","protected":false},"author":73,"featured_media":669,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-670","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/posts\/670","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/comments?post=670"}],"version-history":[{"count":0,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/posts\/670\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/media\/669"}],"wp:attachment":[{"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/media?parent=670"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/categories?post=670"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/struct.ai\/articles\/wp-json\/wp\/v2\/tags?post=670"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}