Tempo 3.1 adds safer Kafka ingest and trace redaction
Tempo 3.1, the distributed tracing release described by Grafana Labs on 2026-09-30, brings several changes that matter beyond one product: safer ingestion, more accurate trace-based metrics, stricter handling of sensitive data and faster reads from stored traces. Distributed tracing means following a request as it passes through different services, so teams can see where time is spent and where failures appear.
A major operational change is in Kafka-based ingestion. Kafka is a message system used to move data between software components, and Tempo 3.0 moved microservices-mode ingestion onto Kafka. In that earlier release, the client could authenticate only with SASL PLAIN, a username-and-password method, and only over an unencrypted connection. Tempo 3.1 adds TLS, which encrypts data in transit, and supports SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER and AWS_MSK_IAM. For teams sending traces across networks or into managed Kafka-compatible systems, the broader authentication and encryption options reduce the need to weaken broker settings just to accept trace traffic.
The release also adds rack-aware fetching through a new client_rack option. Rack-aware fetching, based on KIP-392, lets consumers read from a replica in their own availability zone when brokers have rack IDs configured. The practical issue is cost and latency: Tempo previously fetched from the partition leader wherever it lived, so reading traces could cross zone boundaries when the leader was elsewhere. Grafana Labs notes that this is a read-side setting for block-builders, live-stores and metrics-generators, not for the distributor that writes records to Kafka. Tempo 3.1 also makes producer compression configurable. The distributor can now use none, gzip, snappy, lz4 or zstd. If no option is set, the Kafka client uses snappy by default. Grafana Labs says this matters for Kafka-compatible backends that accept only one codec, citing Azure Event Hubs, which supports the Kafka protocol but accepts only gzip.
The most security-relevant change is query-based trace redaction. Redaction means permanently removing data from stored traces. Tempo 3.0 added redaction, but tempo-cli redact accepted only trace IDs, so operators had to list every affected trace. Tempo 3.1 lets operators submit a TraceQL query instead. TraceQL is Tempo's query language for trace data. Grafana Labs gives an example of matching a span attribute containing leaked personal information and recommends starting with --dry-run, which counts what would be removed without changing blocks. The command prints a batch ID and job count, while per-tenant metrics for job counts and matched or removed traces appear on the Redaction row of the Backend Work dashboard included with Tempo's monitoring mixin.
That extra power comes with risk. Grafana Labs states that running without --dry-run rewrites blocks and cannot be undone. To reduce accidental deletion, the accepted query syntax is deliberately limited for now to a single spanset filter with equality comparisons against resource.* and span.* attributes, combined with && and ||. Operators can also use --start and --end to restrict the time range, using now, an offset such as now-7d or an RFC3339 timestamp. This can make jobs faster and more efficient in high-volume installations because Tempo pauses compaction for a tenant while redaction is applying. Grafana Labs warns that these time range options should be used only after every scheduler and worker in a cell is running at least Tempo 3.1, because an older worker ignores the window and removes every query match in each block it receives, with no error and no recovery path.
Tempo 3.1 also changes TraceQL metrics, which are ad hoc metrics queried directly from trace data. Sampling-aware metrics queries add an experimental with(extrapolate=true) hint. Sampling means storing only part of the trace stream, for example 50 percent of traces. Without correction, a rate() query over 50 percent sampled data reports about half the underlying traffic. The new hint reads the sampling rate from the OpenTelemetry probability sampling field in tracestate, when present, and scales spans accordingly. At 50 percent sampling, each stored span counts as two, so a query matching 1,000 spans reports 2,000. Spans without a recorded sampling rate count as one, which makes partial adoption safe. Grafana Labs says the hint requires vParquet4 blocks or later and applies to rate, count_over_time, sum_over_time, avg_over_time, histogram_over_time, quantile_over_time and compare, but not to min_over_time or max_over_time.
TraceQL metrics can now use arithmetic operators +, -, * and / between two metrics queries, so an error-rate calculation can divide an error rate query by a total traffic query inside TraceQL itself. Each sub-query needs parentheses, and scalars can be used on either side, such as multiplying a duration quantile by 1000 to express it in milliseconds. When queries are grouped, series combine only when their label sets match exactly; a side with no labels is broadcast across every series on the other side. The extrapolate hint can be applied to the whole arithmetic expression, but Grafana Labs says placing with(...) inside a sub-query is a syntax error.
Metrics queries should also become faster as new data is written. The span-only fetch path introduced experimentally in Tempo 3.0 is now enabled by default for vParquet5 blocks, the default block format in Tempo 3.1. Apache Parquet is a column-based storage format designed for efficient analytical reads. Queries that do not need full trace structure can process individual spans instead of whole traces, reducing latency and memory use. Grafana Labs reports that in one warmed-up test, a simple { } | rate() query with span-only fetch enabled completed in slightly over half the time of the same query with it disabled. Operators can opt out per tenant or per query.
The metrics-generator, an optional Tempo component that turns ingested spans into RED metrics (rate, errors and duration) and service graphs, also receives several updates. A new unmatched_span_kind label on the expired edges counter helps explain missing service-map links by showing which side of a client-server span pair failed to arrive. Tempo 3.1 also recognises the OpenTelemetry semantic convention change from db.system to db.system.name, accepting both for identifying database requests and naming virtual nodes. If both attributes are present, db.system takes precedence, avoiding dashboard and alert renames after software development kit upgrades. Grafana Labs also says the span-metrics and service-graphs processors now borrow pooled label buffers instead of allocating new ones each time, with steady-state service-graphs benchmarks reporting zero allocations per operation.
For reading individual traces, Tempo 3.1 adds experimental span pruning on the trace-by-ID v2 endpoint.