Prompt × Context × Harness·Production Incident·15 min

The Prompt Injection Attack

An external knowledge page tells your Agent to access customer records. The requested tools are real and currently allowed. Keep the workflow useful without treating untrusted content as authority.

Experience

Work the problem before reading the explanation.

INCIDENT 003 · SECURITY

The dangerous instruction arrived as data.

You own the Agent's trust and capability boundary.

A support Agent reads external knowledge while answering refund questions. One retrieved page includes instructions asking it to enrich the answer with full CRM data and send account identifiers through an allowed tool path. A second attack is subtler. A benign security article also contains attack-like wording.

Objective

Block malicious instruction flow and sensitive tool reach while keeping useful external knowledge available and avoiding excessive false positives or human review.

Stakes

A prompt can be untrusted data even when it looks like an instruction.

In one sentence

The model cannot be the sole security boundary: trust, capabilities and irreversible actions must be enforced by the runtime around it.

Loading Mission Engine…

Reflection

Turn the outcome into a rule you can reuse.

ENGINEERING DEBRIEF

Prompt injection is a trust-boundary failure, not just a wording problem.

Models cannot be the sole security boundary. Trust, capabilities and sensitive actions must be enforced by the harness/runtime.

The Agent saw system policy, developer workflow instructions and external content in one working context. Without provenance-aware trust and capability enforcement, untrusted data could reach real CRM tools. Denying all external content removes the attack but destroys useful work and creates false positives. Defense in depth keeps useful data readable while enforcing what it is allowed to cause.

—Data can contain instruction-shaped text without gaining instruction authority.
—Least privilege limits blast radius even when model behavior is wrong.
—Sensitive actions need runtime gates that do not depend on the model agreeing with the policy.
—Security evaluation must include benign lookalikes, not only obvious attacks.

Key takeaways

Trust has provenance

System policy, developer intent, user requests and retrieved data do not have equal authority.

Capabilities are a runtime decision

A model request is not permission to export customer data.

Overblocking is also failure

A policy that denies every external-content workflow is safe only by making the product useless.

Learn More

Connect the experience to concepts, references, and transfer.

LEARNING CONTEXT

Learning context

Instruction-shaped text does not gain authority merely because a model can read it; capability policy belongs at the runtime boundary.

Not seen

Mental models

S00-M04Runtime enforcement vs model persuasion
S01-M01Instruction authority and provenance
S04-M03Capability boundary and least privilege
S04-M05Human approval at the high-risk boundary
S06-M05Trust boundary and defense in depth for untrusted context
S07-M01Traceability as causal execution history
S07-M04Outcome vs trajectory evaluation

Suggested backfill

No shipped prerequisite is required before entering this incident.

Transfer the model

The same malicious instruction arrives inside a tool result instead of a retrieved document. Which defenses still hold?

View full learning path

Carry the idea into another problem or build.

Final Boss preview: Ship the Production Support Agent

Retrieval, tool reliability and security now collide in one release decision.

Open the integrated build