CORE GUIDE
Model routing and fallback
Production LLM applications should choose models by task, risk and current system conditions, with explicit fallbacks when the preferred route is unavailable, too slow, too expensive or insufficiently reliable.
Mental model
Routing is a policy that selects a model or execution path from observable task features and operational signals. Fallback is a bounded alternative with known quality and capability differences, not an emergency switch to any available model.
Why it matters
A single-model architecture hides trade-offs until incidents occur. Different models vary in latency, cost, tool support, context limits and behavior. Routing lets the application spend capability where it matters, while fallback keeps service available without silently changing guarantees when the preferred dependency fails.
01
Route by requirements, then bound fallback behavior
Define task classes and the minimum capabilities each requires: structured output, tool use, latency ceiling, context size, cost or quality floor. Select the preferred route, then predefine fallbacks that still satisfy the hard constraints. If no fallback can preserve a critical guarantee, degrade explicitly, queue the task or ask for human handling instead of pretending equivalence.
02
Example: fast chat model cannot replace tool-capable model
A customer-support app routes simple FAQ to a fast cheap model and account changes to a tool-capable model. During an outage, FAQ can fall back to a second small model, but account changes are queued because the fallback lacks the required tool contract. Availability is preserved without silently widening or weakening behavior.
Common failure modes
- Routing only by model price while ignoring capability and consequence.
- Switching to a fallback that cannot satisfy the same output or tool contract.
- Changing models during incidents without measuring quality, latency and error distribution shifts.
Engineering heuristics
- Define hard capability constraints before optimizing cost.
- Treat fallback quality and feature loss as part of the product contract.
- Observe route selection, fallback frequency and outcome quality in production.
Takeaways
- 01Routing is an application policy, not a model feature.
- 02A fallback is valid only when its capability loss is understood and bounded.
- 03Explicit degradation is safer than silent model substitution.
Reading evidence
This records actions you actually took; it does not claim mastery, proficiency, or certification.
Used in
This Concept is reused across these canonical learning paths.