CORE GUIDE
SHIP / BLOCK / INCONCLUSIVE
Release evaluation should end in an explicit decision state: SHIP when evidence clears required gates, BLOCK when a veto or unacceptable regression is present, and INCONCLUSIVE when evidence is too weak to decide safely.
Mental model
These three states separate product quality from evidence quality. BLOCK means evidence is strong enough to show a problem; INCONCLUSIVE means the system has not earned either confidence to ship or proof to reject.
Why it matters
Teams often force every evaluation into pass/fail, which encourages shipping on noisy wins or blocking on weak signals. An inconclusive state creates room to collect more data. Explicit vetoes also prevent strong aggregate scores from averaging away safety, cost or slice regressions that should stop release.
01
Synthesize evidence through gates and sufficiency checks
Define required quality floors, critical vetoes and evidence-sufficiency criteria before evaluation runs. First apply non-negotiable vetoes, then check whether observed improvements or regressions are large enough relative to uncertainty, and only then decide SHIP or BLOCK. If evidence width, sample size or judge reliability cannot support the conclusion, return INCONCLUSIVE and specify what evidence is missing.
02
Example: average score rises while safety slice collapses
A model update improves the aggregate score by four points, but a safety-critical refund slice drops below its release floor. The correct outcome is BLOCK, not SHIP with a warning. In another run, the aggregate gain is one point with a very small sample; no veto fires, but evidence is too weak, so the outcome is INCONCLUSIVE.
Common failure modes
- Reducing every evaluation to one aggregate pass threshold.
- Treating no detected regression as proof that shipping is safe.
- Using INCONCLUSIVE as a vague escape instead of naming the missing evidence.
Engineering heuristics
- Define vetoes before seeing candidate results.
- Separate evidence sufficiency from candidate quality.
- Make every INCONCLUSIVE decision name the next evidence needed.
Takeaways
- 01SHIP, BLOCK and INCONCLUSIVE are evidence states, not sentiment labels.
- 02Critical vetoes should not be averaged away.
- 03Insufficient evidence is a legitimate result that should drive the next experiment.
Reading evidence
This records actions you actually took; it does not claim mastery, proficiency, or certification.
Used in
This Concept is reused across these canonical learning paths.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.