Written by: Nimesh Chakravarthi, Co-founder & CTO, Struct
Key Takeaways for a Production Zipkin Pipeline
- A production Zipkin pipeline routes spans through an OpenTelemetry Collector with 1–5% tail sampling, then persists to Elasticsearch and exposes the UI behind nginx with authentication.
- Baseline MTTR, trace coverage, and alert volume should be measured before implementation so the pipeline’s 80% triage-time reduction can be quantified.
- Log-trace correlation via injected
trace_idfields lets Struct and engineers pivot directly from alerts to full traces without manual searching. - Continuous monitoring of Collector metrics such as dropped spans and sampling ratios prevents silent pipeline degradation and keeps storage costs predictable.
- Let Struct handle first-pass investigations — connect your Zipkin traces to Struct and let AI complete the initial analysis before your engineer is fully awake.
1. Define Objective and Current State
Start by auditing three numbers: trace coverage (percentage of services that emit spans), weekly alert volume, and mean time to resolution (MTTR). Teams that skip this step routinely over-instrument low-value services while leaving critical payment or auth flows dark.
Map every service to one of three states: uninstrumented, head-sampled only, or tail-sampled with persistent storage. Services in the first two states are the ones that extend MTTR during incidents because they produce incomplete or missing trace data when failures occur. That is why instrumentation priority should follow blast radius, since services that affect the most users or revenue when they degrade are the ones where incomplete traces cost the most investigation time.
Establish baseline MTTR now. The goal of the pipeline described below is to cut active triage time by 80%, turning a 45-minute investigation into a 5-minute review. That delta is only measurable if you record the starting point.
Quantify MTTR improvements with Struct — connect your Zipkin traces to Struct and let AI complete the first-pass investigation before your engineer is fully awake.
2. Step-by-Step Implementation
OTel SDK instrumentation. Instrument each service with the OpenTelemetry SDK for its language. The Go SDK’s TracerProvider uses AlwaysSample in development, which is appropriate for development but must be replaced in production. Set the sampler via environment variable to avoid code changes at deploy time:
# service environment (all languages) OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.05 OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 ParentBased sampling propagates the root span’s sampling decision to all child spans, keeping full traces intact rather than producing orphaned child spans with no parent.
OTel Collector gateway configuration (otel-collector-config.yaml). Deploy the Collector as a Kubernetes Deployment in a gateway pattern so all nodes funnel spans through a single pipeline. The Collector receives telemetry via receivers, processes it through configurable pipelines, and exports to one or more backends.
# otel-collector-config.yaml (2026 defaults) receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 zipkin: endpoint: 0.0.0.0:9411 # ingest legacy Zipkin-format spans processors: memory_limiter: check_interval: 1s limit_mib: 512 tail_sampling: decision_wait: 10s num_traces: 50000 policies: - name: errors-policy type: status_code status_code: {status_codes: [ERROR]} - name: slow-traces-policy type: latency latency: {threshold_ms: 1000} - name: probabilistic-policy type: probabilistic probabilistic: {sampling_percentage: 2} exporters: zipkin: endpoint: http://zipkin:9411/api/v2/spans elasticsearch: endpoints: [https://es-host:9200] index: otel-traces tls: ca_file: /certs/ca.crt service: pipelines: traces: receivers: [otlp, zipkin] processors: [memory_limiter, tail_sampling] exporters: [zipkin, elasticsearch] Tail sampling evaluates complete traces after all spans are collected and applies configurable policies to retain those with errors, high latency, or other criteria of interest. This approach delivers higher-quality datasets than head-based sampling because decisions are made after the full trace is assembled. The probabilistic_sampler processor controls remaining volume, so ingestion costs stay predictable.
Zipkin on Elasticsearch (zipkin-deployment.yaml).
# zipkin-deployment.yaml (2026 defaults) apiVersion: apps/v1 kind: Deployment metadata: name: zipkin spec: replicas: 2 template: spec: containers: - name: zipkin image: openzipkin/zipkin:3 env: - name: STORAGE_TYPE value: elasticsearch - name: ES_HOSTS value: https://es-host:9200 - name: ES_USERNAME valueFrom: secretKeyRef: {name: es-creds, key: username} - name: ES_PASSWORD valueFrom: secretKeyRef: {name: es-creds, key: password} ports: - containerPort: 9411 nginx reverse proxy with basic auth (nginx-zipkin.conf). API gateways, which function as specialized reverse proxies, act as a single control point providing centralized token validation, rate limiting, TLS termination, and centralized logging, whereas traditional reverse proxies primarily handle the core network functions such as TLS termination and routing.
# nginx-zipkin.conf (2026 defaults) server { listen 443 ssl; server_name zipkin.internal.example.com; ssl_certificate /etc/ssl/zipkin.crt; ssl_certificate_key /etc/ssl/zipkin.key; ssl_protocols TLSv1.2 TLSv1.3; add_header Strict-Transport-Security "max-age=63072000" always; location / { auth_basic "Zipkin Production"; auth_basic_user_file /etc/nginx/.htpasswd; proxy_pass http://zipkin:9411; proxy_set_header X-Request-ID $request_id; } } Zero-trust principles recommend a default deny-all stance and explicitly allowing only necessary communication. Pair this nginx config with a Kubernetes NetworkPolicy that blocks direct pod-to-pod access to port 9411 from outside the observability namespace.
Log-trace correlation. Inject trace_id and span_id into every structured log line using the OTel SDK’s logging bridge. This single field is what allows Struct and on-call engineers to pivot from a log line directly to the full trace without manual searching.
See trace correlation in action — once your pipeline is live, Struct ingests traces and correlated logs to generate a root-cause dashboard in under 5 minutes.
3. How This Pipeline Fits into Engineering Operations
The five-layer pipeline you just deployed, SDK, Collector, storage, proxy, and correlation, produces high-fidelity trace data. A trace pipeline that terminates at a Zipkin UI is only half the solution. The operational value appears when traces feed alerting channels and incident handoff workflows.
Configure Zipkin or your Elasticsearch index as a data source in your alerting stack such as Grafana, Datadog, or Prometheus with the spanmetrics connector generating RED-style metrics from trace spans. Set SLO-based alerts on p99 latency and error rate derived from span data. When those alerts fire into Slack or PagerDuty, the trace_id embedded in the alert payload gives Struct the anchor it needs to pull the full trace, correlated logs, and relevant code context automatically.
For incident handoff, include the Struct dashboard URL in every post-incident review. The dynamically generated timeline, which merges trace spans, log events, and deployment markers, replaces the manual reconstruction that typically consumes the first 30 minutes of a post-mortem.
Standardize incident handoffs with Struct — encode your team’s specific escalation paths and runbook steps directly into Struct so every alert follows the same investigation procedure, regardless of who is on call.
4. Measurement and Improvement of Your Trace Pipeline
A tracing pipeline that is not monitored will silently degrade. The six metrics below represent the minimum viable monitoring surface, since they catch three failure modes that cause silent data loss and three cost or effectiveness issues that surprise teams at month-end.
The failure modes include dropped spans, backend failures, and sampling misconfiguration. The cost and effectiveness issues include unbounded ingestion, retention drift, and investigations that do not improve MTTR. Use this table as a checklist for day-one instrumentation.
| Metric | Source | Alert Threshold | What It Indicates |
|---|---|---|---|
otelcol_processor_dropped_spans |
OTel Collector | > 0 sustained for 5 min | Pipeline backpressure or memory limit hit |
otelcol_exporter_send_failed_spans |
OTel Collector | > 0 | Elasticsearch or Zipkin backend unreachable |
otelcol_receiver_accepted_spans |
OTel Collector | Drop > 20% week-over-week | Services stopped emitting spans |
| Tail sampling decision ratio (sampled / evaluated) | OTel Collector tail_sampling processor | Error-policy rate drops to 0 | Error spans not being retained |
| Elasticsearch index size growth rate | Elasticsearch cluster API | > 20 GB/day (adjust per retention policy) | Sampling rate too high or ILM policy misconfigured |
| MTTR delta (pre/post pipeline) | Incident tracker | No improvement after 30 days | Traces not correlated to alerts or Struct not ingesting |
Track your pipeline’s ROI — Struct surfaces MTTR trends and investigation quality scores so engineering leadership can quantify the pipeline’s impact on reliability.
5. Common Pitfalls and Storage Decisions
Over-sampling. AlwaysSample is recommended only for development or initial setup because it retains 100% of spans regardless of their diagnostic value. Leaving it enabled in production generates storage costs that scale linearly with traffic, since every request becomes a stored trace, and it creates Elasticsearch index pressure that degrades query performance during incidents, exactly when fast trace lookup matters most.
Missing trace context in logs. If trace_id is absent from log lines, the correlation between a log-based alert and its originating trace is manual. Enforce trace_id injection at the logging framework level, such as Logback MDC or Python logging filters, as a non-negotiable instrumentation standard.
No pipeline monitoring. Tail sampling introduces memory and timing considerations described in the implementation section above, so poorly chosen policies can drop valuable traces or overwhelm the pipeline. Treat the Collector as a production service, expose its Prometheus metrics endpoint, alert on dropped spans, and include it in your incident runbook.
Elasticsearch vs. Cassandra storage decision. Zipkin supports storage backends including Cassandra and Elasticsearch, each with distinct operational trade-offs. The table below maps five decision dimensions to concrete recommendations so your team can decide whether query patterns, write volume, and existing expertise justify Cassandra’s complexity or whether Elasticsearch’s query flexibility and ecosystem integration deliver better return.
| Dimension | Elasticsearch | Cassandra | Recommendation |
|---|---|---|---|
| Query flexibility | Full-text search, ES|QL aggregations, approximate PERCENTILE via TDigest | Primary-key lookups, limited ad-hoc queries | Elasticsearch for teams needing ad-hoc trace search |
| Write throughput | High, scales horizontally with shards | Very high, optimized for write-heavy workloads | Cassandra for extreme write volume (>100k spans/sec) |
| Operational complexity | Moderate, ILM policies manage retention | High, requires tuning compaction and TTLs | Elasticsearch for teams without Cassandra expertise |
| Cross-cluster resilience | External/out-of-cluster storage survives cluster failures | Same benefit when deployed externally | Both, deploy outside the application cluster |
| Ecosystem integration | Native Kibana, Grafana, Struct ingestion | Requires separate query layer for dashboards | Elasticsearch for teams already using the ELK stack |
6. Tie-in to Automated Investigation Tools
A hardened Zipkin pipeline, OTel SDK to Collector with tail sampling to Elasticsearch to nginx auth, produces high-fidelity, cost-controlled trace data. That data becomes operationally transformative only when it is consumed automatically at alert time rather than manually at investigation time.
Struct integrates directly into Slack and PagerDuty alerting channels. When an alert fires, Struct queries the Elasticsearch-backed Zipkin index using the trace_id embedded in the alert, correlates the trace spans with CloudWatch or Datadog logs, maps the timeline against recent GitHub deployments, and delivers a root-cause dashboard to the Slack thread before the on-call engineer has finished reading the alert. The investigation that previously took 30–45 minutes of manual tool-hopping is completed in under 5 minutes, matching the 80% reduction established as the pipeline’s baseline goal.
For teams with custom runbooks, Struct’s composable widget system encodes those procedures directly. Specific correlation ID formats, escalation paths, and service-specific diagnostic queries are all executed automatically on every alert, giving junior engineers the same starting context that previously required a senior engineer’s tribal knowledge.
Frequently Asked Questions
What is the minimum observability maturity required before this pipeline adds value?
Teams need at least structured logging with a consistent request ID field, one or more services emitting OTel or Zipkin-format spans, and an alerting channel such as Slack or PagerDuty. The pipeline described here is designed to harden and extend an existing dev-grade setup, not to replace basic logging. If your services emit no spans at all, start with OTel auto-instrumentation for your primary language before deploying the Collector gateway.
How long does a production rollout realistically take?
For a team with existing Kubernetes infrastructure and a managed Elasticsearch cluster, the Collector deployment, Zipkin backend, and nginx proxy can be production-ready in one to two sprint cycles, roughly two to four weeks. OTel SDK instrumentation across all services typically takes longer and should be prioritized by blast radius. Struct’s integration with the resulting pipeline takes under 10 minutes once the Elasticsearch index is live and trace_id is present in log lines.
How does this architecture address security and compliance requirements?
The nginx reverse proxy enforces TLS 1.2 or higher and HTTP authentication, which prevents unauthenticated access to the Zipkin UI and its raw span data. Kubernetes NetworkPolicies block direct pod-to-pod access to the Zipkin port. Elasticsearch credentials are stored as Kubernetes Secrets and injected at runtime. For SOC 2 and HIPAA environments, Struct processes logs and traces ephemerally and holds SOC 2 and HIPAA certifications. Teams with strict requirements that prohibit any data leaving their VPC should evaluate on-premise deployment options before integrating any external tooling.
Can junior engineers safely use the output of this pipeline during an on-call incident?
Yes, and this is one of the primary operational benefits. When Struct consumes the trace pipeline and delivers a pre-built root-cause dashboard to the Slack alert thread, a junior engineer’s starting point matches a senior engineer’s starting point. They see a correlated timeline, blast radius assessment, and suggested fix. They do not need to know which Elasticsearch index to query or how to interpret raw Zipkin span JSON.
The Slack-native conversational interface also allows them to ask follow-up questions such as “pull logs from 5 minutes before the spike” or “check if this affects user segment X” without leaving the incident thread or requiring escalation.
What is the right tail sampling rate for a high-traffic production service?
A practical starting point is a probabilistic fallback rate of 1–5% combined with policy-based retention for all error spans and all traces exceeding 1,000 ms latency. This combination ensures that the traces most relevant to incidents are always captured while keeping Elasticsearch index growth predictable. Monitor the Collector’s dropped-span and accepted-span metrics weekly for the first month and adjust the probabilistic rate based on actual storage consumption and the ratio of error traces to total traces. Teams with strict SLAs should bias toward retaining more error traces even at higher storage cost.
Conclusion
A production-grade Zipkin pipeline requires five layers working in concert. These layers are OTel SDK instrumentation with ParentBased sampling configured via environment variables, an OTel Collector gateway applying tail sampling policies that always retain errors and slow traces, a Zipkin backend persisted to Elasticsearch deployed outside the application cluster, an nginx reverse proxy enforcing TLS and authentication, and log-trace correlation via injected trace_id fields. Each layer addresses a specific failure mode that causes 3 a.m. incidents to drag on, including incomplete traces, unbounded storage costs, unauthenticated UI exposure, and the inability to pivot from a log line to its originating trace.
The pipeline described here is the prerequisite, not the destination. Once high-fidelity, cost-controlled traces flow into Elasticsearch, the operational leverage comes from automating the investigation layer that consumes them. Struct ingests those traces alongside logs and code context, completes the root-cause analysis in under 5 minutes, and delivers the result directly to the Slack thread where the on-call engineer is already working.
Set up Struct and streamline on-call — configure Struct in under 10 minutes and let AI handle your next on-call investigation from first alert to suggested fix.
