Evaluation Failure Lab
Agent v2 looks better on the dashboard. Your job is not to tune the agent—it is to debug the evaluation process and decide whether the evidence is strong enough to ship.
Evaluation is a release-decision system, not a single score: representative slices, evidence strength, veto conditions, judge coverage, and production economics determine whether an apparent improvement is trustworthy.
The team improved retrieval and tool behavior in a customer-support agent. A demo-heavy evaluation says v2 wins. The hidden problem: long-horizon and safety-critical refund cases regressed.
—
—
—
—
—
—
—
—
The aggregate is a weighted mixture. Change the dataset preset to see which failures the evaluation gives enough importance to influence the release decision.
| Slice | Weight | v1 | v2 | Δ |
|---|
Synthetic cost units. The cost gate is intentionally independent from quality.
Why can a better benchmark score produce a worse release decision?
AI systems rarely fail uniformly. A candidate can improve common questions while regressing on long-horizon tasks, rare tool sequences, or safety-critical actions. When the evaluation compresses all of those cases into one average, the most important failure can disappear inside a strong headline number.
Representative data beats convenient demos
A demo-biased set overweights the cases the team already knows how to solve. A production-like set increases the influence of difficult and consequential cases. This lab uses fixed synthetic slice scores so the learner can isolate the evaluation design itself from model randomness.
Thresholds and vetoes answer different questions
A global threshold asks whether average performance is high enough. A veto asks whether a specific failure is unacceptable regardless of the average. If an AI system can perform an irreversible refund, a safety regression may deserve a hard release gate even when the rest of the system improves.
More samples reduce uncertainty; they do not fix bad coverage
The modeled evidence width shrinks as the sample size grows. That demonstrates why small evaluations provide weaker evidence. But collecting more examples from a biased distribution does not make the evaluation representative. Sample size and dataset design are independent engineering decisions.
Judge strategy changes coverage, noise, and cost
Deterministic checks are stable and cheap, but can miss semantic or process-level defects. Rubric judges can cover more nuanced quality but introduce judge noise and cost. Mixed evaluation combines complementary checks. None of these choices is universally correct; the evaluation method must match the failure modes that matter.
What is simulated here?
All slice scores, confidence behavior, judge noise, and cost values are synthetic educational quantities. No live LLM judge, benchmark, customer-support traffic, or repeated stochastic trial is being executed. Real release policies require representative data, validated graders and verifiers, and organization-specific risk criteria.
Common questions
No. Aggregate metrics are useful summaries, but they should not override critical regressions or evidence-quality requirements.
It improves evidence strength only if the cases are representative and the scoring method measures the behavior you care about.
No. Vetoes are appropriate for selected non-negotiable conditions. Too many hard gates can make iteration impossible.
Because engineering evidence can be insufficient. A release process should be allowed to say “we do not know yet” instead of forcing every noisy comparison into ship or block.
Define the minimum release gate for a refund-capable support agent.
Choose the dataset mix, evidence strength, judge strategy, safety boundary, and economic constraint you would require before deployment. The goal is a defensible policy—not the highest possible dashboard score.