CORE GUIDE
Evaluate retrieval before generation
Retrieval evaluation asks whether the system surfaced the necessary evidence, at usable rank and under the right authority constraints, before judging the generated answer.
Mental model
Separate retrieval from generation: given a query and expected evidence, measure whether the retrieval pipeline returned the right source set before the model had a chance to compensate or hallucinate.
Why it matters
End-to-end answer quality can hide retrieval defects. A powerful model may answer from prior knowledge even when the retriever missed the source, while a weak generator can make good retrieval look bad. Retrieval-specific evaluation isolates recall, ranking and authority failures so the correct subsystem is improved.
01
Build query–evidence judgments and slice them
Create representative queries with one or more acceptable evidence targets, including exact-ID, paraphrase, stale-source and permission-sensitive cases. Measure recall at candidate and final-context stages, ranking metrics when order matters, and whether controlling sources outrank superseded material. Inspect misses by slice to decide whether the fix belongs in chunking, indexing, query rewriting, filters or reranking.
02
Example: the final answer is right for the wrong reason
A model correctly states a 14-day refund window even though retrieval returned only the old 30-day policy, because the model happened to know the newer rule. End-to-end grading passes, but retrieval evaluation fails the query because the controlling evidence never reached context. That failure matters because a future policy change may not exist in model memory.
Common failure modes
- Evaluating only final answers and assuming correct output proves correct retrieval.
- Using random queries that do not represent difficult retrieval slices.
- Counting any relevant source as success even when a stale source outranks the controlling one.
Engineering heuristics
- Maintain labeled query–evidence pairs for critical knowledge domains.
- Measure candidate recall separately from final-context precision.
- Include authority, freshness and permission cases in retrieval test suites.
Takeaways
- 01Good generation cannot excuse missing evidence.
- 02Retrieval metrics localize failures before the model stage.
- 03Authority-sensitive retrieval needs more than semantic relevance labels.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.