RAG has multiple failure points
Retrieval-Augmented Generation (RAG) combines search or retrieval with generation. A poor answer may be caused by retrieval, ranking, missing source content, prompt construction, or generation.
Test retrieval separately
Before judging the generated answer, verify whether the system retrieved the right source material.
- Relevant documents retrieved
- Critical document not missed
- Irrelevant documents controlled
- Correct version used
- Permissions respected
- Ranking is useful
Test groundedness
Groundedness asks whether claims in the answer are supported by the retrieved authoritative context.
Test answer relevance and completeness
A grounded answer can still fail if it does not answer the user's actual question or omits a critical condition.
Test citations and source attribution
If the application provides citations, validate that cited sources actually support the associated claims and that the user can distinguish authoritative sources from generated interpretation.
Test missing-information behaviour
Include cases where the knowledge base does not contain the answer. A trustworthy RAG system should avoid fabricating unsupported detail.
Test permission boundaries
Retrieval must respect authorization. Validate users cannot retrieve or indirectly infer content outside their permitted scope.
Build a RAG evaluation matrix
| Layer | Measure |
|---|---|
| Retrieval | Recall, relevance, correct document/version |
| Context | Ranking, completeness, noise |
| Generation | Groundedness, relevance, completeness |
| Experience | Citations, clarity, refusal/fallback behaviour |