CORE GUIDE
Slices and regression suites
Aggregate quality is not enough: release evaluation needs stable slices that represent important users, tasks and failure modes, plus regression tests that prevent solved problems from returning.
Mental model
An evaluation suite is a portfolio of slices. Each slice represents a meaningful operating condition, and regression cases preserve previously discovered failures as permanent release evidence.
Why it matters
A model upgrade can improve the overall score while breaking multilingual queries, long contexts or high-impact tool calls. Random samples may underrepresent exactly the cases a product cares about. Explicit slices make trade-offs visible, while regression cases convert incidents and bugs into durable organizational memory that future releases must respect.
01
Turn product risk into named evaluation slices
Define slices from user segments, task families, difficulty, language, tool patterns, context size and known risk boundaries. Give critical slices their own thresholds instead of relying only on a weighted average. When a production or evaluation failure is understood, add a representative regression case with the expected evidence and outcome so the same defect is caught before future releases.
02
Example: aggregate score improves while Chinese support regresses
A new model raises overall support accuracy from 91% to 93%, but the Chinese billing slice falls from 92% to 82%. A release policy with a minimum per-slice threshold blocks the rollout until the regression is understood, even though the headline metric improved.
Common failure modes
- Using one aggregate score as the only release criterion.
- Creating dozens of slices with too few examples to support any decision.
- Fixing an incident without adding a regression case that protects the learned invariant.
Engineering heuristics
- Choose slices that correspond to real product decisions or risks.
- Maintain explicit veto thresholds for critical slices.
- Promote understood incidents and high-value bugs into durable regression cases.
Takeaways
- 01Average improvement can hide important regressions.
- 02Slices connect evaluation to product risk.
- 03Regression suites preserve lessons across model and application changes.
Used in
This Concept is reused across these canonical learning paths.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.