CORE GUIDE
Sampling controls
Temperature, top-p and related decoding controls reshape how the runtime samples from model probabilities; they change variability, not the model's underlying knowledge or guarantees.
Mental model
Sampling controls are runtime policies over a next-token distribution. They can concentrate or broaden which tokens are likely to be selected, but they do not repair missing evidence, incorrect context or weak model capability.
Why it matters
Teams often tune temperature as a universal quality knob. Lower randomness can make output look consistent while preserving the same systematic error, and higher randomness can surface alternatives while increasing variance. Correct tuning depends on whether the task values repeatability, exploration, diversity or calibrated uncertainty.
01
Separate model scores from decoding policy
Start with the model's next-token probabilities, then apply the configured decoding transformation before sampling. Temperature rescales relative logits, while nucleus-style sampling restricts selection to a probability mass. The exact provider surface varies, but the engineering question is stable: which output diversity is acceptable for this task, and what downstream validation absorbs the remaining variance?
02
Example: extraction versus brainstorming
A structured invoice extractor benefits from constrained output and low generation variance because downstream code expects a stable schema. A naming brainstorm benefits from broader sampling because near-duplicate answers have low value. Neither setting tells you whether the model read the correct invoice or whether the names satisfy trademark constraints; those require separate evidence and checks.
Common failure modes
- Using temperature to compensate for irrelevant or missing context.
- Assuming lower temperature means factually correct output.
- Applying one decoding configuration to every task regardless of acceptable variance.
Engineering heuristics
- Tune decoding against task-level evaluation, not aesthetic preference.
- Use deterministic validators where correctness can be checked independently.
- Keep exploration settings separate from production paths that require repeatable contracts.
Takeaways
- 01Sampling changes selection behavior, not truth.
- 02Different tasks need different variance budgets.
- 03Decoding policy and correctness validation are separate concerns.
Reading evidence
This records actions you actually took; it does not claim mastery, proficiency, or certification.
Used in
This Concept is reused across these canonical learning paths.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.