CORE GUIDE
Preference post-training
Preference post-training shapes which behaviors a model tends to choose among plausible continuations, but it does not turn preference signals into hard application guarantees.
Mental model
After broad pretraining, post-training can optimize a model toward preferred responses using demonstrations, rankings, critiques or reward signals. It changes behavioral tendencies and interaction quality, while runtime policy still owns permissions, evidence and irreversible decisions.
Why it matters
Users experience the post-trained model rather than the raw pretraining objective, so helpfulness, instruction following, refusal behavior and response style are strongly affected by this stage. But the optimization target is still a learned preference signal. A model can behave well in common cases and fail under distribution shift, conflicting instructions or unusual tools, which is why product safety cannot be outsourced to alignment behavior alone.
01
Optimize preferred behavior after broad capability learning
Post-training starts from a pretrained model and exposes it to examples or comparisons that represent preferred behavior. Training may directly imitate high-quality demonstrations or optimize against a learned or rule-derived preference signal. The result changes which responses are more likely under similar prompts, but the learned policy remains probabilistic and inherits the coverage limits of the preference data and evaluation process.
02
Example: helpful refusal is behavior, not authorization
A post-trained assistant may usually refuse a request to reveal private customer data. That tendency is valuable, but a production support agent should still lack the underlying permission to export arbitrary records. If an unusual prompt bypasses the learned refusal, runtime least privilege continues to protect the system.
Common failure modes
- Treating aligned conversational behavior as a security boundary.
- Assuming preference data covers every future tool, language and edge case.
- Using post-training to encode dynamic business policy that needs immediate updates.
Engineering heuristics
- Evaluate post-trained behavior on the actual product slices and adversarial boundaries that matter.
- Keep authorization, source-of-truth state and irreversible-action controls outside the model.
- Use post-training for stable behavioral tendencies, not rapidly changing facts or policy.
Takeaways
- 01Post-training shapes behavior after broad capability learning.
- 02Preference optimization changes probabilities, not runtime authority.
- 03Product guarantees still require application-layer evidence and enforcement.
Used in
This Concept is reused across these canonical learning paths.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.