The core idea
Distributed tracing turns "P95 is bad" into "this span is the bottleneck" so you fix the right thing.
P95 latency was bad—but nobody knew where. The team assumed it was the LLM. Or retrieval. Or the network. We added distributed tracing across retrieval, embedding, rerank, and LLM. The trace waterfall revealed the real culprit. Fixing that span moved the needle.
Anonymized but real
Names and identifying details are removed. The process and outcomes are preserved.
Executive summary
The client had an LLM pipeline with high P95 latency. Without tracing, the team was guessing—LLM? Retrieval? Rerank? We instrumented the pipeline with distributed tracing. Each stage (retrieval, embedding, rerank, LLM) became a span. The trace waterfall showed which spans drove P95. The real bottleneck wasn't where they thought. Fixing that span—and following LLM observability best practices—moved the needle.
This is usually discovered after teams feel high P95 latency in production. Use Latency & Serving to connect tracing, serving posture, and the downstream optimization plan.
The situation
Before tracing:
- P95 latency: High—but no visibility into which stage caused it
- Assumptions: Team guessed LLM or retrieval—often wrong
- Fixes: Scattered—optimizing the wrong stage didn't help
What we did
We added distributed tracing across the pipeline:
- Retrieval span: Time for vector/BM25 search
- Embedding span: Time for query embedding
- Rerank span: Time for cross-encoder reranking
- LLM span: Time for generation (TTFT + total)
Each request produced a trace. We aggregated by P95 to see which spans contributed most to tail latency.
Trace waterfall: before/after
Before: The trace waterfall showed that the rerank stage was the dominant contributor to P95—not the LLM. Under load, rerank calls were serialized and queued; tail latency spiked.
After: We batched rerank calls, added a cache for repeated queries, and increased concurrency. The rerank span dropped. P95 improved. The LLM span was never the bottleneck.
This pattern—trace first, fix the right span—is how LLM observability and distributed tracing for LLM systems drive targeted fixes.
Why this worked
Without tracing, the team optimized the wrong thing. With tracing, we knew exactly which span to fix. Observability turned guessing into targeted action.
Next steps
If your LLM pipeline has high P95 and you don't know where, an AI system audit can add tracing and identify the real bottleneck. We instrument pipelines, analyze trace waterfalls, and deliver prioritized fixes after clarifying the symptom on the latency pain page.
P95 high but don't know where?
We add distributed tracing to LLM pipelines, analyze trace waterfalls, and identify the real bottleneck. LLM observability is the operating layer; high P95 latency is the symptom most teams feel first.
Lead magnet
Latency Budget Worksheet — Allocate latency budgets across pipeline stages. Request it.
What made this hard
Instrumenting the pipeline without adding latency—tracing overhead had to be minimal.
What made this work
Trace waterfall analysis by P95—seeing which spans drove tail latency.
Need LLM observability?
If your LLM pipeline has high P95 and you don't know where, our AI audit adds tracing and identifies the bottleneck.
Last updated
February 1, 2026
