CORE GUIDE
Online monitoring
Offline evaluation predicts release quality; online monitoring checks whether real traffic, dependencies and user behavior continue to match those assumptions after deployment.
Mental model
Monitoring is a live comparison between expected operating envelopes and observed production evidence. It should detect drift in inputs, outputs, latency, failures, cost and business outcomes without reducing system health to one average score.
Why it matters
AI behavior can change after release even when the model artifact is unchanged: traffic mix shifts, knowledge becomes stale, upstream APIs fail, prompts evolve and users discover new edge cases. Teams need production signals that connect symptoms to traces and evaluation slices so they can decide whether to continue, degrade, roll back or investigate.
01
Monitor slices tied to release hypotheses
Start from the risks and acceptance criteria used before release. Emit stable semantic events, trace critical model/tool transitions, and track input/output quality proxies, failures, latency and cost by meaningful slice. Set alerts on actionable boundaries and preserve examples for diagnosis. Monitoring should trigger a decision process, not automatically retrain or block based on noisy metrics.
02
Example: average success stays flat while one region fails
A support agent's global resolution rate looks unchanged, but a new policy rollout causes failures for one regional plan. Slice-aware monitoring catches the localized increase in escalations and wrong-source retrieval. The team blocks further rollout for that segment while keeping unaffected traffic live and collecting traces for diagnosis.
Common failure modes
- Watching only aggregate averages and hiding important slices.
- Creating many alerts with no owner or associated response decision.
- Using online user behavior as unquestioned ground truth for model quality.
Engineering heuristics
- Connect every alert to a release risk and an operational response.
- Monitor distributions and critical slices, not only global means.
- Retain representative traces so alerts can be diagnosed rather than merely acknowledged.
Takeaways
- 01Production evidence tests release assumptions continuously.
- 02Monitoring needs actionable slices and ownership.
- 03Online signals complement rather than replace controlled evaluation.
Reading evidence
This records actions you actually took; it does not claim mastery, proficiency, or certification.
Used in
This Concept is reused across these canonical learning paths.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.