RAG Wrong Answers Triage: Is It Recall, Reranking, or Context Construction?

When a RAG system gives a wrong answer, most teams do the same thing: tweak prompts, increase k, add more documents, hope it improves. That's expensive roulette—and it usually fails because the fix was aimed at the wrong layer.

The right move is to classify first, then fix. In production, wrong RAG answers almost always fall into one of three buckets:

• Recall failure: you didn't retrieve the right source
• Ranking failure: you retrieved it, but it was ranked too low or dropped before the model saw it
• Context construction failure: you retrieved okay-ish content, but assembled a misleading context (dupes, contradictions, wrong sections)

This guide gives you 12 signals to classify the failure in minutes, the minimum logging schema you need, and the fix order that yields the highest ROI. No more guesswork.

The 3-bucket model (use this before you "optimize")

Before you touch prompts, k, or rerankers, run every wrong answer through this lens. The wrong bucket leads to the wrong fix—and weeks of wasted effort.

Bucket A — Recall failure (you missed the right thing)

The right doc exists in your corpus, but retrieval never surfaced it. Common causes: embedding mismatch, chunk granularity, query-doc gap, or missing hybrid/lexical search.

Bucket B — Ranking failure (you got it, but buried it)

The right doc is in the candidate set, but doesn't make it into the final context sent to the model. Reranker mis-calibration, selection heuristics, or context budget rules are usually to blame.

Bucket C — Context construction failure (you assembled a misleading context)

Chunks are correct-ish but stitched poorly: duplicates, contradictions, missing constraints, or wrong sections. The model gets confused—and so do users.

Fast triage decision tree (60 seconds)

Use this flow before you touch prompts or "increase k." It tells you which layer is actually failing: recall, ranking/selection, or context construction.

The 12 signals (triage checklist)

Use these signals when inspecting a wrong answer. Each one points toward recall, ranking, or context construction—and tells you where to look in your traces next.

Signal 1 — "No citation" or citation is unrelated

If your answer cites nothing or cites content that doesn't support the claim, it's likely recall or context construction. Quick check: does any retrieved chunk contain the key constraint the user asked about?

Signal 2 — The answer contains confident specifics not present in the retrieved text

If numbers, dates, policy exceptions, or guarantees appear with no evidence in context → context construction or generation overreach. Fix direction: output validation + "evidence-first" answer formatting + citation verification.

Signal 3 — Top retrieved chunks are thematically close but miss the exact requirement

Example: user asks "refund after 30 days," retrieval pulls "refund policy" but not the clause about 30 days. That's often recall failure due to: embedding mismatch, chunk granularity mismatch, missing query rewrite.

Signal 4 — Retrieved chunks include the exact answer, but it's not in the final prompt context

If your pipeline does multi-step retrieval (retrieve → rerank → select), this is classic ranking failure or selection heuristics bug. Tell: "We saw it in logs, but it didn't make it to the model."

Signal 5 — You see duplicated chunks or repeated paragraphs in context

Dupes consume context window and suppress diversity → context construction failure. Common causes: bad dedupe strategy, overlapping chunking, retrieval returns near-duplicates.

Signal 6 — Context contains contradictory chunks (two versions of the policy)

Contradictions force the model to guess → context construction failure or freshness issue. Fix direction: doc versioning + freshness routing, conflict detection, "most recent policy wins" logic.

Signal 7 — Wrong answers cluster around specific intents or document types

If failures spike for "pricing," "legal," "troubleshooting," or "multi-step procedures," that's a clue the system is not robust across cohorts. Often indicates chunking strategy mismatch (doc-type specific).

Signal 8 — Wrong answers spike after content updates

If changing docs breaks answers, you likely have: stale index, missing re-embedding pipeline, caching returning old retrieval results. That's a freshness variant of recall/context issues.

Signal 9 — The model's answer is correct for a similar question but wrong for a more specific one

Often recall: embeddings pull generic chunks, missing specific exception clauses. Fix direction: hybrid search, query expansion, structured filters (product, region, plan tier).

Signal 10 — Increasing k improves recall but makes answers worse

If higher k helps find relevant bits but answer quality drops, you're in context construction failure: too much context, irrelevant chunks increase confusion, contradictions. Fix direction: tighter selection + reranking, context size budgets, chunk quality scoring.

Signal 11 — Reranking improves relevance but cost/latency explodes

This indicates your reranking is applied too broadly. It's not "reranking is bad," it's "reranking needs routing." Fix direction: route reranking only when retrieval confidence is low or the intent is high-risk.

Signal 12 — The retrieved evidence supports the correct answer, but the model still answers wrong

This is where generation is the culprit (not retrieval): instruction conflicts, formatting/policy constraints, model choice, lack of "cite-or-refuse" behavior. Fix direction: prompt hierarchy cleanup + output validation + "refuse if not evidenced" rules.

What to log to diagnose in minutes (minimum schema)

Triage is only as fast as your traces. If you want to classify failures in hours instead of weeks, log these per request:

• Query + intent label (if you have it)
• Candidate retrieval list: doc ids, chunk ids, similarity scores
• Final selected context: chunk ids + ordering
• Reranker scores (if used)
• Prompt hash/version + model id
• Answer + citations produced
• Validation outcomes (schema pass/fail, citation checks)
• Latency breakdown per stage
• Cost breakdown (tokens, rerank calls, tool calls)

This is enough to classify recall vs ranking vs context construction reliably.

The fix order (highest ROI first)

Fixing the wrong layer burns time and budget. When teams fix RAG systematically, this sequence usually delivers the best ROI:

If you skip #1, you'll burn time on "optimizations" that don't move the needle.

Mini case studies (what fast triage unlocks)

Note: numbers are illustrative. The structure is the point—baseline by cohort, controlled changes, and distribution-aware before/after evidence.

A simple self-test you can run today (10 queries)

Don't wait for perfect instrumentation. Pick 10 real production queries that users marked wrong—or that you know are wrong—and answer these four questions for each:

• Did the correct source exist in the corpus? (Y/N)
• Was it retrieved in top N? (Y/N)
• Was it selected into final context? (Y/N)
• Did the answer cite it correctly? (Y/N)

By the end, you'll see which bucket dominates. That single insight will tell you where to focus first—and save you from chasing the wrong fix.

The triage worksheet (10 queries, 10 minutes)

Turn this into a repeatable process. Use a simple worksheet that forces the 4 yes/no questions for each wrong answer—and captures the bucket label and fix direction. Here's the structure:

Next steps