Instrumenting Distributed Tracing Without Blowing the Latency Budget

Every span you create, every context you propagate, every byte you serialize to send to a collector costs CPU time and adds wall-clock latency to the request path. Most teams pretend this isn't true, but it is, and I've spent enough hours staring at flame graphs after a bad rollout to know exactly where the cost hides. Deciding how much you're willing to pay is the job, and proving in production that you didn't pay more than you budgeted comes next.

Sampling comes first, and it's a fight you have before you write any code

Head-based sampling decides at the start of a trace whether to record it. You flip a weighted coin, tag the context, every downstream service honors that decision. Cheap, predictable, usually a single conditional check per span. I like it for exactly one reason: you know the cost up front, and it doesn't move.

Head-based sampling is blind, though. It has no way of knowing a request is about to hit a slow database query or blow up three services downstream. So you get a sample that looks statistically clean and tells you almost nothing about the requests you'd actually lose sleep over: the slow ones, the failed ones, the ones a customer is screenshotting right now.

Tail-based sampling waits until the full trace assembles before deciding what to keep, which addresses the blindness problem. Jaeger's tail sampling processor does this, and so does Grafana Tempo. Both buffer spans for a window, often several seconds, before making the call. That buffering isn't free; you need memory to hold spans in flight and, in practice, a dedicated collector tier to do the aggregation. Honeycomb has written honestly about what this costs at scale, and the lesson repeats everywhere it's tried: the collector layer becomes its own capacity-planning problem, separate and apart from the services you're actually tracing. Nobody budgets for that the first time, but everyone budgets for it the second time.

Once teams have been burned once, what most production systems actually run is a hybrid. Cheap head-based sampling at a low rate, often well under 10%, gives you baseline visibility. Then a rule layer forces a 100% keep on errors, timeouts, or anything crossing a latency threshold. It's the same logic as log-level filtering, just applied to traces instead of text. Bias your sampling toward the traces that would cost you the most to miss, rather than spreading it evenly across traffic.

Span cardinality is where the real damage happens

Nobody's latency problem comes from the act of creating a span. It comes from what gets bolted onto that span afterward. I've watched this play out almost the same way three separate times: a team instruments a service, feels good about coverage, ships it, and six months later the tracing backend is timing out on queries because somebody tagged every single span with a raw user ID, or a hashed request body, or a full URL with the query string still attached.

High-cardinality attributes don't just bloat storage, though they do that too. They slow the export path, because most collectors and backends index or aggregate span tags at ingest. Slap a UUID on every span and you've defeated that aggregation completely; the backend now treats each span as effectively unique, and that multiplies write amplification downstream in a way that's genuinely painful to unwind later. OpenTelemetry's semantic conventions exist partly to head this off. They push you toward a bounded, known set of keys (http.status_code, db.system, rpc.method) instead of letting every team invent its own tagging scheme from scratch. Following the convention functions as a cardinality control, even though it presents itself as a style guide.

So set a cap. Decide the max number of distinct values per attribute key before instrumentation ships, not after a bill shows up or a query starts timing out and forces the conversation. If a value can take on unbounded distinct values, user IDs, session tokens, free-text input, it belongs in a log line correlated to the trace ID rather than as a span attribute. Traces carry structure; logs carry detail. Mixing the two up is, in my experience, the single most common reason instrumentation gets ripped out and redone a year later.

You cannot reason your way to a latency number

You have to measure it, using the same service and the same load pattern, with tracing on versus tracing off, as a real comparison.

The measurement also has to isolate tracing as the only variable that moved. Run the load test twice: once against a build with the tracing SDK fully removed (not sampled to zero, because a no-op tracer still walks different code paths than code that was never compiled in), and once with tracing live. Then look at p50, p95, and p99, not the average. Tracing overhead shows up disproportionately in the tail, because context propagation and span export compete with the request thread for CPU time under load, and that contention gets worse exactly when the system is already under stress, which is exactly when you need the trace the most. A service that adds two milliseconds at p50 might add twenty at p99. Average those together and you've hidden the only number that mattered.

Export mechanism matters as much as instrumentation density here, maybe more. Synchronous export, where the request thread blocks waiting to hand spans to the collector, is a latency budget killer, full stop, and I won't defend it under any circumstance. Batched, asynchronous export through something like the OpenTelemetry Collector's OTLP exporter, run as a sidecar or local agent, gets the network call off the request path entirely. What's left after that decoupling is almost entirely span creation and attribute serialization, both measurable, both controllable through the cardinality discipline above.

Google's Dapper paper, still one of the foundational documents in this whole field, reported sampling and instrumentation overhead in the low single digits as a percentage of production traffic, at Google's scale. That number has held up remarkably well as an industry rule of thumb for what "acceptable" looks like. If your measurement comes back meaningfully above that, the instrumentation needs rework before it ships, rather than getting waved through as the cost of doing business.

Write the budget down. An actual number, not a feeling.

None of the above works without a stated latency budget attached to the service before anyone writes instrumentation code. Something like: this endpoint has a 200ms p99 SLO, tracing gets 3ms of that, no more.

Once that number exists, every sampling decision and every cardinality call becomes a tradeoff against a fixed constraint instead of an open-ended argument about how important observability is (it is important; that was never the argument). Teams that skip this step over-instrument early, when the system is small and the cost is invisible, and then face a brutal retrofit once traffic grows and the overhead compounds on itself. The teams that get this right treat the trace as a production feature with its own performance contract, load-tested and regression-tracked like any other code on the request path.

Tracing earns its keep by making incidents faster to find and faster to close. Staying light enough to not become the incident itself is the only way it does that job.