Written by: Nimesh Chakravarthi, Co-founder & CTO, Struct
Key Takeaways for Using Tempo in Production
- Grafana Tempo stores traces in object storage and uses TraceQL for fast span-level queries, which suits production incident response.
- Configure Alloy with resilient queue and retry settings so it buffers traces and prevents span loss during Tempo restarts or high load.
- Apply tail-based sampling to retain 100% of error and slow traces while cutting healthy trace storage by up to 90–95%.
- Enable Tempo’s metrics-generator for RED metrics with exemplars and configure Loki derived fields so logs link directly to traces.
- Struct ingests correlated signals from Tempo and posts root cause analysis to Slack; see how Struct automates your incident response
Buffering Tempo Writes with Alloy / OpenTelemetry Collector
Running the OpenTelemetry Collector or Grafana Alloy as a sidecar or DaemonSet decouples your services from Tempo’s write path. When Tempo slows down or restarts, the collector absorbs the burst instead of dropping spans at the SDK level.
Use these queue and retry settings for Alloy’s otelcol.exporter.otlp block:
otelcol.exporter.otlp "tempo" { client { endpoint = "tempo:4317" tls { insecure = true } } sending_queue { enabled = true num_consumers = 8 queue_size = 10000 } retry_on_failure { enabled = true initial_interval = "5s" max_interval = "30s" max_elapsed_time = "300s" } }
Set queue_size to at least 10,000 spans for services that emit more than 5,000 spans per second, so the buffer absorbs burst traffic when Tempo is slow. Increase num_consumers to match available CPU cores on the collector pod, which lets the collector drain that queue in parallel. This buffering matters for Struct because it reads traces directly from Tempo’s HTTP API, and dropped spans in the collector never reach Tempo for correlation against a firing alert.
Tail-Based Sampling for Error-Focused Retention
With the collector properly buffered to prevent span loss, you can apply tail-based sampling confidently. Head-based sampling makes decisions before a trace completes, so errors and slow traces are discarded at the same rate as healthy ones. Tail-based sampling waits for the full trace, then applies policy rules that keep 100% of error traces and a configurable percentage of healthy ones.
A practical policy for a service emitting 50,000 traces per minute at $0.30 per GB stored:
| Policy | Condition | Keep Rate | Estimated Storage Impact |
|---|---|---|---|
| Error traces | status = error |
100% | Baseline (errors are typically <5% of volume) |
| Slow traces | duration > 2s |
100% | +5–10% of total volume |
| Healthy traces | All others | 5–10% | Reduces healthy-trace storage by 90–95% |
At 5% effective sampling on healthy traces, a team storing 500 GB per month of raw traces can reduce that to roughly 50–75 GB per month after accounting for 100% retention of errors and slow spans. Configure this in the otelcol.processor.tail_sampling block in Alloy, and set decision_wait: 30s so late-arriving spans complete before the policy evaluates. Struct’s investigation engine benefits directly because 100% of error traces remain available, so Struct does not hit gaps when correlating a Sentry exception to a Tempo trace ID.
Using Metrics-Generator for RED Metrics and Exemplars
Tempo’s built-in metrics-generator derives Rate, Error, and Duration (RED) metrics from ingested spans without a separate instrumentation pass. Enable it in tempo.yaml:
metrics_generator: registry: external_labels: source: tempo storage: path: /var/tempo/generator/wal remote_write: - url: http://prometheus:9090/api/v1/write processor: service_graphs: dimensions: [http.method, http.status_code] span_metrics: dimensions: [http.method, http.status_code, http.target] enable_target_info: true
The service_graphs processor emits a directed graph of service-to-service call rates and error rates as Prometheus metrics. The span_metrics processor emits per-operation histograms. Both processors attach exemplars, which are trace IDs embedded in Prometheus metric samples, so a Grafana panel that shows a p95 latency spike links directly to a representative trace with one click. Struct ingests these exemplar-linked metrics from Prometheus and uses the embedded trace IDs to pull the exact Tempo trace that caused the anomaly, which removes the manual “find a trace from that time window” step.
Linking Loki Logs to Tempo Traces
Loki derived fields create clickable trace ID links inside log lines. Configure them in the Loki datasource in Grafana:
derivedFields: - name: TraceID matcherRegex: "trace_id=(\w+)" url: "$${__value.raw}" datasourceUid: tempo-uid urlDisplayLabel: "Open in Tempo"
Any log line that contains trace_id=<id> renders a button that opens the full trace in Tempo. For structured JSON logs, use matcherRegex: '"traceId":"(\w+)"'. The investigation phase, which includes identifying the component causing an outage, consumes 60–80% of MTTR in distributed systems, so removing manual copy and paste between Loki and Tempo directly attacks that majority. Struct reads both Loki logs and Tempo traces and uses the same trace ID linkage to build a unified incident timeline automatically.
TraceQL Patterns for Errors and Slow Spans
TraceQL is Tempo’s purpose-built query language for span-level filtering across millions of traces. TraceQL supports filtering by service name, span duration, status codes, HTTP methods, and custom attributes, and it returns results incrementally through streaming search instead of waiting for a full dataset scan.
Use these TraceQL patterns during production incidents:
| Scenario | TraceQL Query | When to Use |
|---|---|---|
| All error traces | { status = error } |
First query on any alert |
| Slow checkout spans | { .service.name = "checkout" && duration > 2s } |
p95 latency alert on checkout |
| Auth-to-payment failures | { .service.name = "auth" } && { .service.name = "payment" } |
Cross-service dependency failures |
| HTTP 5xx on specific route | { .http.status_code >= 500 && .http.target = "/api/order" } |
Route-level error spike |
TraceQL allows multiple conditions in a single query, which narrows traces across large datasets during failure investigations in distributed systems. Struct encodes these patterns into its runbook engine, so when a checkout latency alert fires, Struct runs the relevant TraceQL query and surfaces the slowest spans in its generated dashboard without manual input.
Connecting Tempo Traces to Pyroscope Profiles
Grafana Pyroscope continuous profiling integrates with Tempo through the profileTypes link in the trace view. Enable this by setting a matching profileURL in the Tempo datasource and ensuring your services emit profiles with the same service.name attribute used in spans. When a slow span appears in Tempo, a “Profiles” button opens the CPU or memory flame graph for that service during the trace’s time window.
This workflow narrows root cause from “the checkout service was slow” to “a specific function consumed 94% of CPU during that request.” Struct uses the Pyroscope API to attach profile snapshots to its incident report when a trace shows anomalous duration, which gives engineers a code-level answer without switching tools.
Tempo Production Readiness Checklist
| Area | Setting | Recommended Value | Status |
|---|---|---|---|
| Collector | Queue size | ≥ 10,000 | ☐ |
| Collector | Retry max elapsed time | 300s | ☐ |
| Sampling | Error trace retention | 100% | ☐ |
| Sampling | Healthy trace retention | 5–10% | ☐ |
| Metrics-generator | Service graphs enabled | true | ☐ |
| Metrics-generator | Exemplars enabled | true | ☐ |
| Loki | Derived fields regex | trace_id pattern set | ☐ |
| Pyroscope | profileURL linked | Matching service.name | ☐ |
Common Tempo and Struct Issues
| Symptom | Likely Cause | Fix |
|---|---|---|
| “No traces found” for a known error | Head-based sampling dropped the trace | Switch to tail-based sampling with 100% error retention |
| Exemplar links return 404 in Grafana | Trace ID in exemplar not ingested by Tempo | Verify collector queue is not dropping spans, then check Tempo ingester logs |
| TraceQL query times out | Querying too large a time window | Narrow the window to 1 hour or less and use streaming search |
| Derived field link does not appear in Loki | Regex does not match log format | Test the regex against a raw log line in Grafana Explore |
| Metrics-generator metrics missing | Remote write endpoint unreachable | Check the Prometheus remote write URL and network policy |
Tempo vs. Jaeger in Production Environments
The configurations above prepare Tempo for production incident response and automated investigations. Teams that evaluate tracing backends often compare Tempo with Jaeger, which is the other major open-source option. Jaeger’s native storage primarily uses Cassandra, Elasticsearch, or OpenSearch as distributed backends, which require cluster management and tend to scale vertically. Tempo writes directly to object storage such as S3, GCS, or Azure Blob, which separates compute from storage and reduces operational overhead.
For tail-based sampling, Jaeger’s collector supports it through the jaeger-agent pipeline, but the configuration is less composable than Alloy’s processor chain. Tempo’s metrics-generator also has no direct Jaeger equivalent, so RED metrics from Jaeger require a separate Prometheus instrumentation layer.
| Dimension | Grafana Tempo | Jaeger |
|---|---|---|
| Storage backend | Object storage (S3/GCS/Azure) | Cassandra, Elasticsearch or OpenSearch |
| Tail-based sampling | Native via Alloy processor | Requires separate collector config |
| RED metrics generation | Built-in metrics-generator | Requires external instrumentation |
| Query language | TraceQL (span-level filtering) | Jaeger UI search (tag-based) |
Tempo’s native exemplar linking and metrics-generator reduce the manual correlation overhead discussed earlier more directly than Jaeger’s architecture allows. High-performing SRE teams typically remediate major incidents in under an hour, and the configurations in this guide push toward that benchmark by eliminating the manual steps that consume most investigation time. See how Struct helps teams hit that sub-hour MTTR target
Conclusion: Turning Tempo into an Incident Signal Engine
A properly configured Grafana Tempo stack with Alloy queue and retry settings, tail-based sampling at 100% error retention, metrics-generator exemplars, Loki derived fields, and TraceQL incident patterns turns traces into an active incident signal. Major outages carry significant financial costs, so every minute of triage time you remove has direct financial impact.
Struct sits at the end of this pipeline, ingesting the correlated signals Tempo produces and delivering root cause analysis to Slack before the on-call engineer finishes reading the alert. The 3 AM log-hunting expedition becomes a 5-minute review. Book a demo to see this pipeline in action
Frequently Asked Questions
Does Struct work if our Tempo traces are incomplete or inconsistently instrumented?
Struct performs best when trace IDs propagate consistently across services and logs contain matching trace ID fields. If instrumentation is partial, such as only certain services emitting spans, Struct correlates what is available and flags gaps in the investigation report. The tail-based sampling configuration in this guide keeps error traces at 100% retention, which gives Struct the highest-value data even when overall instrumentation coverage is uneven.
Teams with significant instrumentation gaps should add OpenTelemetry SDK instrumentation to their highest-traffic services first, because those services generate the most actionable traces during incidents.
How does Struct handle the volume of traces generated by a high-throughput production service?
Struct queries Tempo’s HTTP API instead of ingesting raw span streams, so raw trace volume does not affect Struct directly. When an alert fires, Struct runs targeted TraceQL queries scoped to the alert’s time window and affected service, which retrieves only the relevant traces.
The tail-based sampling configuration described earlier preserves 100% of error and slow traces, which are the only traces Struct needs during an incident investigation, while dramatically reducing storage costs on healthy traffic. This approach keeps Struct’s query latency low regardless of overall ingestion rate.
Can Struct consume metrics-generator exemplars directly from Prometheus?
Yes. Struct integrates with Prometheus as an observability source. When metrics-generator emits RED metrics with embedded exemplar trace IDs, Struct reads those exemplars during its investigation and uses the trace IDs to pull the corresponding Tempo traces automatically.
This mechanism allows Struct to jump from a p95 latency spike on a Prometheus alert directly to the specific slow trace without any manual lookup. Enabling the span_metrics processor with exemplars, as shown in the configuration above, is the prerequisite for this workflow.
Is there a minimum Grafana Tempo version required for the configurations in this guide?
The metrics-generator with exemplar support requires Tempo 1.4 or later. TraceQL streaming search requires Tempo 2.2 or later. Tail-based sampling through Grafana Alloy uses the otelcol.processor.tail_sampling component. Trace-to-profiles linking with Pyroscope requires a compatible Pyroscope deployment.
Most teams that run Tempo on a current Grafana Cloud stack or a self-hosted deployment from 2024 onward already meet these feature requirements and do not need an upgrade.
How long does it take to connect Struct to an existing Tempo and Grafana setup?
Struct’s initial setup usually takes under 10 minutes. Connecting Grafana as an observability source requires a service account token with read access to Tempo and Prometheus datasources. After connection, Struct immediately begins listening to your configured Slack or PagerDuty alerting channels.
The first automated investigation runs the next time an alert fires. No changes to your existing Tempo configuration are required for basic integration, although enabling the metrics-generator and derived fields described in this guide significantly improves the depth of Struct’s automated root cause analysis.
