The core idea
Wrong answers in RAG usually start before generation. Diagnose retrieval, ranking, context construction, and freshness first. Then prove the fix with a before/after baseline.
When a RAG system starts giving wrong answers, most teams reach for the prompt first. It feels fast, visible, and cheap. In production, that instinct often sends the team into prompt-tweaking roulette.
Many wrong answers are not prompt failures. They come from retrieval misses, weak ranking, bad context assembly, stale indexes, or document version drift. This checklist helps you diagnose the failing layer first, then prove improvement with before/after baselines instead of opinions.
Context
Part of the RAG Reliability Hub. Related reading: Why RAG Still Hallucinates When Retrieval Looks Fine, RAG Recall vs Precision Diagnostic, RAG Wrong Answers Triage, RAG Optimization Service, AI Audit.
1) How to use this checklist
Read each signal and mark it as No, Partial, or Yes.
- No: you can show logs, traces, or eval evidence that this is controlled
- Partial: it happens in some cohorts, tenants, intents, or releases
- Yes: it is clearly happening in production now
Rule: the more Partial and Yes answers you mark, the less you should change the prompt first. This checklist is designed to stop teams from masking retrieval and context failures with superficial instruction edits.
Use a baseline set of real wrong-answer traces before making changes. After every fix, compare before vs after on the same cohort: candidate recall, selection recall, contradiction rate, citation support rate, and answer accuracy.
| Mark | Meaning | What to do next |
|---|---|---|
| No | Controlled and evidenced | Keep monitoring |
| Partial | Cohort-specific or intermittent | Instrument the cohort and validate a targeted fix |
| Yes | Current production failure | Fix the layer before prompt changes |
2) Retrieval & recall signals
1. The right document is not present in the candidate set
If the answer exists in your corpus but the correct document never appears in the top candidates, you have a recall problem. Prompt edits will not recover evidence that retrieval never surfaced.
2. You only use dense search for literal-heavy queries
IDs, SKUs, error codes, version strings, and exact policy names are common failure modes for dense retrieval alone. If your system does not support hybrid retrieval, literal recall is often weaker than teams expect. See the Hybrid Search + Reranking Playbook for how to add BM25 + fusion.
3. Chunking does not match query granularity
Long chunks blur specific exceptions. Tiny chunks lose section meaning. If users ask precise questions against documents chunked for broad semantic similarity, recall and support quality both degrade.
4. The index is stale or mapped to the wrong document version
A good prompt cannot compensate for missing re-indexing. If recently updated docs are absent from retrieval or old chunks still rank highly, wrong answers often start with ingestion hygiene, not generation.
5. Metadata filters are excluding the right source
Wrong product, version, locale, plan, or region filters can silently remove the correct document from the candidate pool. This is common in enterprise knowledge bases with overlapping document families.
6. Query normalization strips literals that matter
Aggressive query rewriting can erase the exact tokens retrieval needs. If user input like ERR-8492 becomes a generic paraphrase, the system may retrieve conceptually similar but wrong material.
What to prove before touching the prompt
Measure candidate recall @N on a fixed wrong-answer cohort. If the right source is missing, fix retrieval first. See the RAG Recall vs Precision Diagnostic.
3) Ranking & selection signals
7. The right document is in candidates but not in the final context
This is the classic ranking or selection failure. Retrieval found the answer, but reranking, dedupe, or selection heuristics dropped it before the model saw it.
8. You do not use a reranker, or the reranker is too weak for the task
If candidate recall is acceptable but final-context relevance is poor, weak or missing reranking is a likely cause. This is especially visible in corpora with many semantically similar documents.
9. Your context budget is too small for the evidence needed
The right chunk can be ranked correctly and still get truncated away. If selection logic optimizes for token budgets without protecting the best evidence, wrong answers become systematic.
10. Duplicate or overlapping chunks crowd out better evidence
Near-duplicates waste context budget, inflate confidence in one source family, and reduce diversity of supporting evidence. The model sees more text, but not more useful proof.
11. Ranking ignores business-critical constraints
If the ranking layer treats version, region, or plan as optional instead of first-class, the final context may look relevant while still being wrong for the user's cohort.
| Signal | Likely layer | What to measure |
|---|---|---|
| Right source in top 50, absent from prompt context | Ranking / selection | Selection recall, reranker hit rate, truncation rate |
| More candidates but weaker final answers | Selection / noise control | Final-context precision, duplicate ratio |
| Specific versions keep losing to generic pages | Ranking with missing constraints | Version-aware win rate |
4) Context construction signals
12. The final context contains contradictions
When the model sees conflicting policy versions, conflicting procedures, or incompatible exception rules, it often guesses. The answer may sound polished while still being wrong.
13. The context omits key constraints
Missing version, region, plan tier, customer segment, or time boundary is one of the most common reasons grounded answers still fail. The context is not useless, just incomplete in the wrong way.
14. Citations do not actually support the claim
Citation presence is not citation support. If the cited chunk does not contain the governing statement, your trust layer is cosmetic and wrong answers will continue to look credible.
15. Heading and section-path information is missing
Raw text alone can blur whether a sentence comes from a limitation, an exception, a deprecated section, or a troubleshooting note. Section path often matters as much as the sentence itself.
16. Context ordering buries the governing rule below weaker evidence
Models are sensitive to what appears first and what appears repeatedly. If generic summaries come before the controlling section, the answer can drift even when the correct rule is technically present.
Evidence rule
If the answer cites a chunk, verify that the chunk supports the claim directly. If not, classify it as a context construction failure first, not a prompt style issue.
5) Freshness & versioning signals
17. The source document was updated, but the index was not
This is the simplest path to confidently wrong answers. Teams often notice it only after support tickets spike because the answer still sounds plausible.
18. Policy or pricing content is stale
Policy and pricing mistakes have disproportionate trust cost. If those documents change frequently, freshness should be treated as a reliability control, not a content operations detail.
19. The system mixes version families such as v1 and v2
Version drift is a common hidden source of hallucination-like behavior. The model is not inventing facts; it is combining evidence from incompatible document generations.
20. Cache invalidation is weaker than document update frequency
Even with a fresh index, stale retrieval caches can keep serving obsolete evidence. That can make the pipeline look healthy in logs while the answer layer keeps citing yesterday's truth.
6) Generation vs retrieval: when the prompt is the problem
Sometimes the prompt really is the issue. That is usually true only after the evidence chain is already correct: the final context is relevant, up to date, internally consistent, and clearly supports the answer.
- Weak cite-or-refuse behavior: the model answers beyond the evidence instead of refusing
- Loose format or validation: the output can overstate certainty, omit caveats, or violate required structure
- Constraint hierarchy is unclear: system, retrieval, and task instructions compete instead of reinforcing grounded behavior
If you cannot prove the context is correct first, treat prompt changes as unvalidated speculation.
7) Scoring: when to escalate
Count only Yes signals for escalation. Use Partial as pressure to instrument sooner.
- 0–2 Yes: monitor, keep cohort-level traces, and resist unnecessary prompt edits
- 3–5 Yes: instrument and fix retrieval, ranking, freshness, or context construction before changing prompts
- 6+ Yes: run a full RAG audit and stop prompt tweaking until you have a trace-backed baseline
| Score | Interpretation | Recommended action |
|---|---|---|
| 0–2 Yes | Local issues or isolated cohorts | Monitor and validate on a fixed set |
| 3–5 Yes | Multiple RAG-layer failures are likely | Instrument deeper and fix retrieval/context layers first |
| 6+ Yes | Systemic reliability risk | Run a full audit, baseline, and before/after validation plan |
The goal is not to collect symptoms. It is to identify which layer is failing, establish a baseline, and prove improvement after the fix. That is the difference between engineering and prompt roulette.
8) Next steps
If this checklist surfaced multiple signals, do not jump straight to prompt revisions. First, run the RAG Recall vs Precision Diagnostic to separate recall from noise. Then use the RAG Optimization Service framework to sequence fixes by impact and validation order.
If you need a broader reliability baseline, start with RAG Wrong Answers Triage for faster classification or an AI Audit for trace-backed before/after proof across quality and trust.
Next step
If your team flagged 6+ signals, stop editing prompts in isolation. Establish a baseline, fix the failing RAG layer, and prove improvement on the same wrong-answer cohort before the next release.
FAQ
Questions readers usually ask next
When should we run this checklist?
Run it when wrong answers or hallucinations rise, after a content migration, after a retrieval or reranking change, after docs are updated, or any time the team starts debating prompt edits without trace evidence. It is most useful before you touch the prompt, not after.
How do we distinguish recall vs ranking vs context construction?
If the right source is missing from the candidate set, it is a recall problem. If the right source is in candidates but not in the final context, it is a ranking or selection problem. If the final context contains contradictions, missing constraints, or weak supporting evidence, it is a context construction problem.
How many signals mean we need a real audit?
Use the score as a decision threshold. With 0–2 Yes signals, keep monitoring. With 3–5 Yes signals, instrument and fix retrieval or context layers first. With 6 or more Yes signals, a full RAG audit is usually warranted because the system is likely failing across multiple layers at once.
When is the prompt actually the problem?
The prompt becomes the likely issue when the final context is correct, recent, consistent, and clearly supports the answer, but the model still answers incorrectly, overstates confidence, or fails to refuse. That usually points to weak cite-or-refuse rules, loose formatting constraints, or missing output validation.
How is this different from RAG Wrong Answers Triage?
RAG Wrong Answers Triage is a faster classification guide for identifying the failing bucket quickly. This checklist is broader and more audit-oriented: it groups production signals by retrieval, ranking, context construction, and freshness, then adds scoring and escalation rules so teams know when to stop tweaking and start measuring.
Need a trace-backed RAG audit?
Start with the Recall vs Precision diagnostic or move directly to an AI Audit if you need before/after proof across production traces.
Last updated
March 7, 2026
