CORE GUIDE
Tokenization
Models process token sequences, not human-visible characters or words, so cost, limits and behavior depend on how text is segmented.
Mental model
Before generation, text is converted into token IDs. The model reasons over that sequence, and context limits are measured in tokens rather than pages or characters.
Why it matters
Tokenization affects context budget, latency, price and sometimes behavior. Code, uncommon names, multilingual text and whitespace can consume very different numbers of tokens even when they look similar in length to a person.
01
From text to model input
A tokenizer maps recurring text fragments to integer IDs. Common fragments may be represented compactly while unusual strings split into many pieces. Those IDs become the sequence processed by the model; generation produces token IDs that are decoded back into text.
02
Example: equal characters, unequal cost
A short identifier, a JSON blob and a Chinese sentence can have similar character counts but different token counts. A system that budgets by characters may unexpectedly truncate instructions or exceed a provider context limit.
Common failure modes
- Estimating context capacity only from word or character counts.
- Ignoring token-heavy logs, code and serialized tool results.
- Assuming token boundaries align with semantic concepts.
Engineering heuristics
- Measure tokens with the tokenizer used by the target model when limits matter.
- Budget separately for instructions, retrieved evidence, tool output and generation.
- Trim low-value serialized data before removing high-value evidence.
Takeaways
- 01Characters are a UI concept; tokens are the model's input accounting unit.
- 02Token count is workload-dependent, not proportional to visible length.
- 03Tokenization should inform budgets, not become a semantic retrieval strategy.
Related concepts from the Knowledge Graph
These relationships come from the canonical graph, not a separate Guide taxonomy.