CORE GUIDE
Failure attribution
Failure attribution identifies which layer—model, prompt, context, retrieval, tool, state, orchestration or runtime—most likely caused an observed bad outcome so the fix targets the mechanism instead of the symptom.
Mental model
Treat an AI outcome as the end of a causal chain. Attribution asks where evidence first diverged from the expected path and which intervention would have prevented that divergence.
Why it matters
Without attribution, teams react to every bad answer by changing the prompt or model. That creates churn and can hide infrastructure, stale data or tool failures. Layered traces and controlled comparisons turn incidents into actionable engineering evidence.
01
Trace the first divergence and test competing explanations
Capture model inputs and outputs, retrieved evidence, tool calls, state transitions and runtime decisions with stable identifiers. Reconstruct the expected path, locate the earliest meaningful mismatch, then change one layer at a time or replay the case under controlled conditions. Mark conclusions as probable when multiple causes remain plausible.
02
Example: hallucinated refund rule is actually stale retrieval
A support assistant cites a refund rule that was removed last week. The team initially blames model hallucination, but the trace shows the retriever returned an old cached policy and the model faithfully summarized it. The correct fix is cache invalidation and source authority, not a stronger 'do not hallucinate' prompt.
Common failure modes
- Assigning every incorrect answer to the model because it is the visible component.
- Looking only at the final response instead of the evidence and state that produced it.
- Changing several layers at once and losing the ability to learn which fix mattered.
Engineering heuristics
- Locate the earliest observable divergence before proposing a fix.
- Keep enough trace evidence to replay important incidents.
- Use controlled interventions to distinguish plausible causes when attribution is uncertain.
Takeaways
- 01A bad output can originate far upstream from generation.
- 02Attribution converts observability into a repair decision.
- 03The best fix targets the causal layer, not the most visible symptom.
Reading evidence
This records actions you actually took; it does not claim mastery, proficiency, or certification.
Used in
This Concept is reused across these canonical learning paths.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.