CORE GUIDE
Evidence granularity
RAG quality depends on retrieving evidence at a granularity that preserves meaning while remaining selective enough for ranking and context budgets; chunk size is therefore a decision, not a magic constant.
Mental model
Granularity defines the unit that retrieval can select and rank. Too coarse, and irrelevant material travels with useful evidence; too fine, and meaning, references or constraints are split across units that no longer stand alone.
Why it matters
Teams often tune top-k or rerankers while leaving chunking arbitrary. But retrieval can only rank the units it is given. Granularity shapes recall, precision, context cost and the ability to cite evidence precisely, so it should be tested against real query and document structure rather than copied from a default.
01
Choose units around semantic boundaries and retrieval tasks
Start with document structure: headings, paragraphs, tables, policies or records. Build candidate units that can answer likely queries without carrying excessive unrelated content, then add overlap only where cross-boundary information is routinely lost. Evaluate with missed-evidence cases, noisy-context cases and citation accuracy rather than optimizing chunk size in isolation.
02
Example: one policy chapter becomes one retrieval chunk
A compliance manual stores each entire chapter as a chunk. A query about one refund exception retrieves the chapter successfully but adds thousands of unrelated tokens, burying the relevant clause. Splitting by policy section improves precision; limited overlap keeps shared definitions available without retrieving the whole chapter.
Common failure modes
- Choosing one global chunk size for every document type.
- Making chunks so small that qualifiers and references lose their meaning.
- Making chunks so large that top-k retrieval becomes context dumping.
Engineering heuristics
- Align chunk boundaries with the semantic unit users actually ask about.
- Evaluate granularity with recall, noise and citation precision together.
- Use overlap to repair boundary loss, not as a substitute for better segmentation.
Takeaways
- 01Retrieval can only rank the evidence units it receives.
- 02Granularity trades semantic completeness against selectivity.
- 03Chunking should be evaluated on real failures, not inherited as a magic number.
Reading evidence
This records actions you actually took; it does not claim mastery, proficiency, or certification.
Used in
This Concept is reused across these canonical learning paths.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.