TL;DR
Tokenization is traditionally viewed as a simple input preprocessing step, but this paper reveals that it secretly acts as a powerful output supervision signal in autoregressive language models. By determining the granularity of the target tokens, tokenization fundamentally changes the difficulty of the learning task and shapes the model's internal representations. This hidden variable means many standard LLM benchmark comparisons are actually comparing models on completely different tasks.
What Problem Does This Paper Address?
When evaluating Large Language Models (LLMs) on tasks like arithmetic, researchers frequently attribute performance differences to a model's underlying "reasoning ability." However, different LLM families use vastly different tokenizers. For example, Llama 3 uses holistic tokenization for numbers, treating a multi-digit integer like 578 as a single token. Conversely, models like Qwen 2 and Mistral use fragmented tokenization, breaking the same number down digit-by-digit into [5, 7, 8].
The authors address a critical conceptual blind spot: tokenization doesn't just change how a model sees inputs; it changes what the model must predict in a single forward pass. Because autoregressive models compute loss over exactly one token at a time, token boundaries define the atomic unit of supervision. The paper investigates how this output granularity alters task difficulty, training dynamics, and the internal representations that emerge within the network.
Methods
To isolate the effects of input representation from output supervision, the authors designed a controlled 2x2 factorial experiment using small decoder-only transformer models trained from scratch on 3-digit addition (e.g., 347 + 231 = 578).
They decoupled tokenization into four distinct model conditions:
- $M_F$: Fragmented input, Fragmented output
- $M_H$: Holistic input, Holistic output
- $M_{Fin, Hout}$: Fragmented input, Holistic output
- $M_{Hin, Fout}$: Holistic input, Fragmented output
Beyond tracking training loss and evaluation accuracy, the authors employed probing classifiers (linear, MLP, and circular probes) trained on the models' residual streams at the = token position. This allowed them to check exactly which digits of the result were internally represented before generation began. Finally, they conducted a literature survey of 120 recent *CL papers to quantify how often the NLP community reports or controls for numeric tokenization.
Datasets
- Synthetic Addition Dataset: 3-digit integer, two-operand addition problems ($x + y = z$, where $x, y \in [0, 999]$ and $z \le 999$). Training problems were uniformly sampled across the result space, with a fixed held-out evaluation set of 500 problems.
- Literature Survey Corpus: 120 papers published in the main or findings tracks of ACL, EMNLP, NAACL, and EACL (2024–2025) focusing on mathematical or numeric reasoning.
Key Insights and Ideas
- Tokenization Dictates the Task: A holistic model must resolve all digits of an answer simultaneously in a single forward pass. A fragmented model only needs to resolve the first digit, deferring subsequent digits to future passes where previous outputs serve as context. They are fundamentally different learning problems.
- Output Trumps Input: A model's training trajectory and final accuracy are almost entirely dictated by its output tokenization scheme, remaining largely invariant to how the inputs are encoded.
- The Minimal Computation Hypothesis: Autoregressive models are computationally lazy. Under fragmented output (in little-endian format), they only compute the specific digit required for the immediate next token, leaving future digits at chance level inside the hidden states.
- Task Demands Can Force Lookahead: When the task structure forces a dependency—such as big-endian addition where carry information propagates from right to left—the model will internally represent future digits even without direct gradient pressure from the immediate loss function.
- A Massive Evaluation Blindspot: The literature survey revealed that 69% of papers compared models across different tokenization regimes without acknowledging it, meaning many benchmarks inadvertently measure task difficulty rather than model capability.
Key Results
- Performance Divergence: Models with fragmented output ($M_F$ and $M_{Hin, Fout}$) converged rapidly and reliably achieved 100% exact match accuracy. Models with holistic output ($M_{H}$ and $M_{Fin, Hout}$) struggled to converge and achieved significantly lower accuracy (~85%).
- Decoupling Proof: As shown below, the performance of $M_{Fin, Hout}$ closely mirrors $M_H$, and $M_{Hin, Fout}$ mirrors $M_F$, proving that output supervision drives the behavior.
- Internal Representations: Probing at the
=sign showed that for fragmented output models, only the immediate next digit (units) was decodable. For holistic models, all three digits became decodable simultaneously by the final layer. - Reporting Gap: Fewer than 10% of surveyed papers explicitly reported the numeric tokenization strategy of the models they evaluated.
Limitations and Future Work
- Scale and Scope: The controlled experiments were limited to small models trained from scratch on 3-digit addition. While the underlying mechanism is scale-independent, the exact magnitude of this effect in frontier LLMs remains unquantified.
- Standard Autoregressive Training Only: The study focuses strictly on standard next-token prediction. It does not explore how output supervision interacts with advanced post-training methods like Reinforcement Learning from Human Feedback (RLHF), process supervision, or Chain-of-Thought (CoT) prompting.
- Probing Constraints: Probing measures geometric decodability rather than the absolute presence or absence of information.
- Future Directions: The authors suggest extending this output supervision framework to other structured domains where tokenization choices vary, such as morphology, date expressions in temporal reasoning, and code generation.
Summary updated