CORE GUIDE
Trust boundaries in AI systems
A trust boundary marks where data, instructions or actions cross between components with different authority, ownership or reliability assumptions.
Mental model
Not all tokens are equal. System policy, user intent, retrieved documents, tool output and external side effects come from different trust domains and must be processed with different authority and validation rules.
Why it matters
AI applications often serialize everything into one context, which can visually flatten the distinction between trusted policy and untrusted content. Retrieved web text can contain instructions, a tool can return malformed data, and a user can request an action they are not authorized to perform. Explicit trust boundaries keep authority in runtime policy instead of letting persuasive text decide what is allowed.
01
Label sources and enforce crossings
Identify boundaries where untrusted or lower-authority information enters a stronger decision domain: user input into policy logic, retrieved content into agent context, model intent into tools, and tool results into durable state. At each crossing define sanitization, validation, permission checks and provenance requirements. The model may reason about the data, but it cannot promote that data's authority by repeating it.
02
Example: instructions inside a retrieved document
A support agent retrieves a troubleshooting page that contains text telling the assistant to export customer records. Retrieval relevance does not grant permission. The document is treated as evidence about troubleshooting only, while runtime tool permissions and system policy prevent the unrelated data-export action.
Common failure modes
- Putting trusted policy and untrusted retrieved instructions into one undifferentiated text block.
- Assuming content becomes trustworthy because it was returned by an approved retrieval system.
- Allowing model interpretation to decide authorization at a side-effect boundary.
Engineering heuristics
- Track provenance and authority separately from semantic relevance.
- Enforce permissions at the tool or runtime layer rather than in natural-language instructions.
- Design validation explicitly at every boundary where state or authority changes.
Takeaways
- 01Trust is a property of source and authority, not fluency.
- 02Retrieval and tool output remain external inputs that require validation.
- 03Runtime boundaries should remain effective even when the model is persuaded by malicious text.
Used in
This Concept is reused across these canonical learning paths.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.