Context × Evaluation·Production Incident·15 min

The Broken RAG Pipeline

The new refund policy is in the knowledge base. Your support agent still answered with the old one. Diagnose the evidence pipeline before this reaches more customers.

Experience

Work the problem before reading the explanation.

INCIDENT 001 · SEV-2

Yesterday's truth shipped today.

You are the on-call AI engineer.

A customer asked whether a purchase from 40 days ago can still be returned. The support agent denied the return and cited an older policy. Operations confirms that a newer policy was published two days ago and is already indexed.

Objective

Restore evidence freshness and grounding without blowing the 6k context, latency, or cost budget.

Stakes

A wrong answer can deny valid refunds at scale.

In one sentence

RAG reliability is a pipeline property: retrieval score, freshness, authority, reranking and context policy must agree on what evidence deserves to reach the model.

Loading Mission Engine…

Reflection

Turn the outcome into a rule you can reuse.

ENGINEERING DEBRIEF

More context was never the real fix.

Retrieval quality is a pipeline property. More context is not the same as better evidence.

The incident was caused by a stale source outranking current authority. Increasing Top-K can improve apparent recall while preserving the wrong source and consuming more context. A robust policy combines retrieval signals with freshness, authority, reranking and a context-retention policy that preserves critical details.

—Similarity is not authority.
—Freshness belongs in retrieval policy, not in a prompt apology after retrieval.
—Aggressive compression can turn a correct evidence set into a poorly grounded answer.
—Trace the evidence path before tuning the model.

Key takeaways

Retrieval is a pipeline

Chunking, candidate retrieval, reranking, freshness and context policy interact.

Authority beats popularity

The highest similarity score should not automatically override the current source of truth.

Observe before tuning

A retrieval trace lets you distinguish missing evidence from stale evidence and context loss.

Learn More

Connect the experience to concepts, references, and transfer.

LEARNING CONTEXT

Learning context

Retrieval quality is a pipeline property: freshness, authority, selection and grounding must agree.

Not seen

Mental models

S02-M04Evidence granularity and chunking
S02-M05Dense, sparse and hybrid retrieval
S02-M06Reranking as a second evidence-selection stage
S02-M07Freshness, authority and source priority
S07-M01Traceability as causal execution history
S07-M03Evaluation environment and verifier design

Suggested backfill

Context Window Lab Context Compression Lab

Transfer the model

The newest document is retrieved, but an older source is legally authoritative. How should the evidence policy decide?

View full learning path

Carry the idea into another problem or build.

Next incident: The $47,000 Retry

The evidence is correct. Now the Agent's tool runtime repeats an irreversible action after a timeout.

Open reliability incident