Fixing Low Recall in Production RAG: +18–30pt Answer Accuracy Without Model Fine-tuning

A production RAG system was returning wrong answers, bad citations, and user feedback like "not helpful." The team assumed the model was the problem. It wasn't. The bottleneck was retrieval—and fixing it moved answer accuracy by 18–30 points without any model fine-tuning.

Executive summary

The client had a RAG-powered Q&A system in production. Answer accuracy was low, citations often pointed to irrelevant or wrong passages, and user feedback was consistently negative. An AI system audit revealed that the LLM was fine—the problem was retrieval. See our RAG Optimization Service for the fix order we use. Relevant passages weren't being retrieved, so the model had nothing to ground on.

This is the technical root cause behind RAG wrong answers. Use RAG Reliability to connect retrieval quality, groundedness, and the commercial impact of bad answers.

We implemented hybrid retrieval (BM25 + vector), added a reranker, fixed chunking to follow document structure, introduced query rewriting, and deduplicated passages. Answer accuracy improved by 18–30 points; retrieval recall@k and nDCG@k improved; grounded answer rate and citation correctness rose.

Baseline (before)

Before any changes, we measured the system as users experienced it:

• Answer accuracy: Low—many answers were wrong or ungrounded
• Citations: Often incorrect or irrelevant
• User feedback: "Not helpful," "wrong information"

We ran an offline evaluation on a golden set of questions with known-good answers. Retrieval recall@k was poor; the right passages were rarely in the top-k.

Diagnosis: why retrieval was missing

We traced the failure modes. The root causes were:

1) Chunking mismatch

Chunks were fixed-size (e.g., 512 tokens) without respect to document structure. Sentences and paragraphs were split mid-thought, and semantic units were broken across chunks.

2) Embedding mismatch

The embedding model didn't align well with the query distribution. Queries were often phrased differently from the source text, and pure vector search missed lexical overlap.

3) No hybrid retrieval

Pure vector search was used. BM25 (lexical) would have caught many queries that vector search missed—especially for technical terms, IDs, and exact phrases.

4) No reranking

Top-k from retrieval went straight to the LLM. A cross-encoder reranker would have reordered results and filtered noise.

The fix

We implemented a structured RAG optimization plan:

• Hybrid retrieval: BM25 + vector search, with reciprocal rank fusion (RRF)
• Reranker: Cross-encoder reranker on top-20 candidates before sending to LLM
• Chunking: Chunk by document structure (sections, paragraphs) instead of fixed token count
• Query rewriting: Expand/paraphrase user queries to improve retrieval coverage
• Deduplication: Merge overlapping or duplicate passages before context construction

Before/After metrics

Why this worked

The LLM was capable—it just needed better context. Hybrid retrieval caught queries that pure vector missed. Reranking improved precision. Chunking by structure preserved semantic units. Query rewriting expanded coverage. Together, these changes gave the model the right passages to ground on.

What we shipped

Artifacts delivered to the client:

• Golden set (184 queries) with human-labeled relevance
• Hybrid retrieval pipeline (BM25 + vector, RRF fusion)
• Cross-encoder reranker on top-20 candidates
• Chunking strategy by document structure (sections, paragraphs)
• Query rewriting rules for expansion/paraphrase
• Deduplication + context construction pipeline
• Retrieval eval harness (recall@k, nDCG@k)
• Before/after benchmark report (exec-ready)

Example artifact: tracing schema snippet

Retrieval eval dashboard (redacted): recall@k, nDCG@k, grounded answer rate.

Retrieval eval dashboard (redacted)

{
  "query_id": "q_abc123",
  "candidates": [{"doc_id": "d1", "score": 0.89}, {"doc_id": "d2", "score": 0.72}],
  "selected_context": ["d1", "d2"],
  "grounded_answer_rate": 0.94,
  "citation_correctness": 0.91
}

Next steps

If your RAG system has low answer accuracy or wrong citations, retrieval is often the culprit. An AI Production Audit or Optimization Sprint engagement can baseline retrieval recall, diagnose root causes, and deliver a prioritized fix plan. If you need to align the buyer on symptoms first, send them to RAG wrong answers.