The core idea
RAG hallucinations often come from context—too long, too noisy, no citation gating—not from the model.
A RAG system was hallucinating—making up facts, citing wrong sources, and mixing information from unrelated documents. The team blamed the model. The real problem was context construction: too much noise, wrong top-k, and no guardrails when evidence was missing.
Anonymized but real
Names and identifying details are removed. The process, metrics, and outcomes are preserved.
Executive summary
The client had a RAG system that produced answers with unsupported claims, wrong citations, and mixed information from different sources. An AI system audit showed that the model was capable of grounding—when given clean, relevant context. The failure was in how context was built and passed to the LLM. See Why RAG Still Hallucinates When Retrieval Looks Fine for the diagnosis and our RAG Optimization Service for the fix order we use.
In buyer language, this usually appears as RAG wrong answers. Use RAG Reliability to tie hallucinations, context quality, and production trust back together.
We implemented context compaction, source filtering, citation gating (cite-only constraint), and a refusal policy when evidence was missing. Groundedness rate and citation coverage improved; unsupported claim rate dropped; user trust (survey) increased.
Baseline (before)
Before changes, we measured:
- Groundedness rate: Low—many answers contained unsupported claims
- Citation coverage: Poor—citations often didn't support the claim
- Context length: Very long—retrieved passages were concatenated without compaction
- Source mixing: Passages from unrelated docs were mixed together
Diagnosis: context construction failures
Common patterns we found:
1) Context too long and dilute
Top-k passages were concatenated raw. Long context diluted the signal; the model "averaged" over noise and sometimes hallucinated to fill gaps.
2) Top-k too large or wrong
Too many passages were passed. Irrelevant passages introduced conflicting information and encouraged the model to "mix" sources.
3) Mixing sources without boundaries
Passages from different documents (e.g., different products, policies) were concatenated. The model couldn't reliably attribute which claim came from which source.
4) No "cite-only" constraint
The prompt didn't enforce that every claim must be backed by a citation. The model could add "helpful" information that wasn't in the context.
5) No refusal when evidence missing
When retrieval returned nothing relevant, the system still tried to answer—leading to pure hallucination.
The fix
We implemented RAG optimization focused on context construction:
- Context compaction: Summarize or extract key sentences from long passages before sending to LLM
- Source filtering: Filter out low-relevance passages; keep only high-confidence retrievals
- Citation gating: Prompt constraint—every factual claim must have a citation; no unsupported claims
- Refusal policy: When retrieval returns no relevant evidence, respond "I don't have enough information" instead of guessing
Metrics
Before/After (validated)
| Metric | Before | After |
|---|---|---|
| Groundedness rate | Low | Significantly higher |
| Unsupported claim rate | High | Dropped |
| Citation coverage | Poor | Improved |
| User trust (survey) | Low | Improved |
Why this worked
The model was capable of grounding when given clean, relevant context. By compacting context, filtering noise, enforcing citation gating, and refusing when evidence was missing, we reduced the conditions that led to hallucination.
What we shipped
Artifacts delivered to the client:
- Context compaction pipeline (summarization + key-sentence extraction)
- Source filtering config (relevance threshold, deduplication rules)
- Citation gating prompt rules (cite-only constraint)
- Refusal policy implementation (when retrieval returns no relevant evidence)
- Groundedness dashboard (ungrounded rate, citation coverage trends)
- Before/after benchmark report (exec-ready)
Example artifact: groundedness scorecard
Eval run on golden set: groundedness and citation coverage before vs after.
| Metric | Before | After |
|---|---|---|
| Groundedness rate | Low | Significantly higher |
| Unsupported claim rate | High | Dropped |
| Citation coverage | Poor | Improved |
Next steps
If your RAG system hallucinates or mixes sources, context construction is often the culprit. An AI Production Audit can baseline groundedness, diagnose context construction issues, and deliver a fix plan. If stakeholders only feel the symptom today, start with RAG wrong answers.
Optimization Sprint
We run Optimization Sprints for RAG systems: baseline groundedness, diagnose context construction, and ship fixes with before/after validation. RAG wrong answers gives the buyer-facing symptom set.
Lead magnet
RAG Retrieval Triage Checklist — Diagnose retrieval and context construction issues. Request it.
What made this hard
Balancing groundedness with helpfulness—refusing when evidence is missing without being unhelpful.
What made this work
Offline eval on groundedness, unsupported claim rate, and citation coverage validated every change.
Need to reduce RAG hallucinations?
If your RAG system hallucinates or mixes sources, our Optimization Sprint baselines groundedness and delivers context construction fixes.
Last updated
February 1, 2026
