CORE GUIDE
Streaming and backpressure
Streaming improves perceived latency only when producers, consumers and downstream parsers agree on flow control; otherwise fast token production can overwhelm slower clients or application work.
Mental model
Backpressure is the contract that prevents a producer from emitting work faster than the next stage can safely consume it. Token streaming is therefore an end-to-end flow-control problem, not just a UI rendering feature.
Why it matters
A model may produce tokens quickly while a client is on a slow network, a parser waits for complete JSON, or a tool pipeline performs expensive work per chunk. If buffers grow without bounds, the product gains memory pressure, stale work and cancellation bugs. Reliable streaming needs framing, bounded buffers and cancellation propagation.
01
Design flow control across the whole stream
Define chunk framing, buffer limits, consumer readiness and cancellation semantics between model provider, server and client. Do not trigger irreversible downstream work from incomplete data unless the protocol supports it. When the consumer slows, either pause upstream work where supported, bound and shed buffered data safely, or terminate with an explicit recoverable state.
02
Example: structured answer streamed to a slow browser
A server streams partial model output while incrementally validating a structured response. The browser disconnects on mobile. Without cancellation propagation the provider keeps generating and the server keeps buffering. With a bounded stream contract, disconnect cancels upstream generation, temporary buffers are released and no half-parsed object is committed.
Common failure modes
- Assuming streaming automatically reduces total latency or cost.
- Using unbounded buffers between model output and slow consumers.
- Committing application state from partial chunks that have not satisfied the output contract.
Engineering heuristics
- Propagate cancellation from client to server to provider when possible.
- Bound buffers and measure queue growth under slow-consumer tests.
- Separate presentation streaming from validated state transitions.
Takeaways
- 01Streaming is an end-to-end protocol.
- 02Backpressure protects the system when consumption is slower than production.
- 03Partial output should not silently bypass validation contracts.
Reading evidence
This records actions you actually took; it does not claim mastery, proficiency, or certification.
Used in
This Concept is reused across these canonical learning paths.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.