CORE GUIDE
Model selection is a workload trade-off
Choose models against the workload and its constraints rather than assuming the largest or newest model is automatically the best choice.
Mental model
Model selection is a Pareto decision across quality, latency, cost, context needs, tool behavior, deployment constraints and failure risk; the best model is the one that satisfies the product contract with acceptable margins.
Why it matters
Benchmarks compress diverse behavior into convenient scores, while production workloads contain specific prompts, languages, tool schemas, latency targets and error costs. A model that leads a broad benchmark can be slower, less stable on your structured output, or unnecessarily expensive. Selection becomes an engineering decision only when candidates are compared on representative tasks and real operating constraints.
01
Evaluate candidates on the actual operating envelope
Build a representative task suite with required quality thresholds, difficult slices and known failure modes. Measure candidate models under the same harness, context policy and tool contracts. Compare not only average task quality but tail latency, variance, cost per successful task, rate limits and recovery behavior. Use routing only when the additional complexity produces measurable value.
02
Example: two models for support classification
A larger model improves classification accuracy by one point but triples latency and cost, while a smaller model already clears the product's error budget. The system keeps the smaller model for routine classification and reserves the larger model for an escalation slice where ambiguity materially changes customer outcomes.
Common failure modes
- Selecting a model from leaderboard rank without workload-specific evaluation.
- Comparing models under different prompts or harness behavior and attributing the difference to the model.
- Ignoring latency variance and cost per successful task.
Engineering heuristics
- Define minimum acceptable quality and operational constraints before comparing models.
- Evaluate difficult workload slices separately from aggregate averages.
- Prefer the simplest model or routing policy that satisfies the contract.
Takeaways
- 01There is no universally best model independent of workload.
- 02Model choice should be tied to measurable product requirements.
- 03Routing complexity must earn its coordination and evaluation cost.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.