When Tokenization is Secretly Output Supervision

Tanja Baeumel, Josef van Genabith, Simon OstermannarXiv 2026 · arXiv

TL;DR

Tokenization is traditionally viewed as a simple input preprocessing step, but this paper reveals that it secretly acts as a powerful output supervision signal in autoregressive language models. By determining the granularity of the target tokens, tokenization fundamentally changes the difficulty of the learning task and shapes the model's internal representations. This hidden variable means many standard LLM benchmark comparisons are actually comparing models on completely different tasks.

What Problem Does This Paper Address?

When evaluating Large Language Models (LLMs) on tasks like arithmetic, researchers frequently attribute performance differences to a model's underlying "reasoning ability." However, different LLM families use vastly different tokenizers. For example, Llama 3 uses holistic tokenization for numbers, treating a multi-digit integer like 578 as a single token. Conversely, models like Qwen 2 and Mistral use fragmented tokenization, breaking the same number down digit-by-digit into [5, 7, 8].

The authors address a critical conceptual blind spot: tokenization doesn't just change how a model sees inputs; it changes what the model must predict in a single forward pass. Because autoregressive models compute loss over exactly one token at a time, token boundaries define the atomic unit of supervision. The paper investigates how this output granularity alters task difficulty, training dynamics, and the internal representations that emerge within the network.

Figure 1: We argue that tokenization in transformer models provides a supervision signal, because it deter- mines the output granularity and thus the task to be solved per forward pass. We support our argument with experimental evidence where we (a) decouple input from output tokenization to demonstrate that differences in (b) learning task difficulty and (c) model internal rep- resentations are due to the output tokenization scheme.

Methods

To isolate the effects of input representation from output supervision, the authors designed a controlled 2x2 factorial experiment using small decoder-only transformer models trained from scratch on 3-digit addition (e.g., 347 + 231 = 578).

They decoupled tokenization into four distinct model conditions:

  1. $M_F$: Fragmented input, Fragmented output
  2. $M_H$: Holistic input, Holistic output
  3. $M_{Fin, Hout}$: Fragmented input, Holistic output
  4. $M_{Hin, Fout}$: Holistic input, Fragmented output

Beyond tracking training loss and evaluation accuracy, the authors employed probing classifiers (linear, MLP, and circular probes) trained on the models' residual streams at the = token position. This allowed them to check exactly which digits of the result were internally represented before generation began. Finally, they conducted a literature survey of 120 recent *CL papers to quantify how often the NLP community reports or controls for numeric tokenization.

Datasets

  • Synthetic Addition Dataset: 3-digit integer, two-operand addition problems ($x + y = z$, where $x, y \in [0, 999]$ and $z \le 999$). Training problems were uniformly sampled across the result space, with a fixed held-out evaluation set of 500 problems.
  • Literature Survey Corpus: 120 papers published in the main or findings tracks of ACL, EMNLP, NAACL, and EACL (2024–2025) focusing on mathematical or numeric reasoning.

Key Insights and Ideas

  • Tokenization Dictates the Task: A holistic model must resolve all digits of an answer simultaneously in a single forward pass. A fragmented model only needs to resolve the first digit, deferring subsequent digits to future passes where previous outputs serve as context. They are fundamentally different learning problems.
  • Output Trumps Input: A model's training trajectory and final accuracy are almost entirely dictated by its output tokenization scheme, remaining largely invariant to how the inputs are encoded.
  • The Minimal Computation Hypothesis: Autoregressive models are computationally lazy. Under fragmented output (in little-endian format), they only compute the specific digit required for the immediate next token, leaving future digits at chance level inside the hidden states.
  • Task Demands Can Force Lookahead: When the task structure forces a dependency—such as big-endian addition where carry information propagates from right to left—the model will internally represent future digits even without direct gradient pressure from the immediate loss function.
  • A Massive Evaluation Blindspot: The literature survey revealed that 69% of papers compared models across different tokenization regimes without acknowledging it, meaning many benchmarks inadvertently measure task difficulty rather than model capability.

Key Results

  • Performance Divergence: Models with fragmented output ($M_F$ and $M_{Hin, Fout}$) converged rapidly and reliably achieved 100% exact match accuracy. Models with holistic output ($M_{H}$ and $M_{Fin, Hout}$) struggled to converge and achieved significantly lower accuracy (~85%).
Training loss and evaluation accuracy across different seeds for standard fragmented ($M_F$) and holistic ($M_H$) models.
  • Decoupling Proof: As shown below, the performance of $M_{Fin, Hout}$ closely mirrors $M_H$, and $M_{Hin, Fout}$ mirrors $M_F$, proving that output supervision drives the behavior.
Training loss and evaluation accuracy for the decoupled input/output models.
  • Internal Representations: Probing at the = sign showed that for fragmented output models, only the immediate next digit (units) was decodable. For holistic models, all three digits became decodable simultaneously by the final layer.
  • Reporting Gap: Fewer than 10% of surveyed papers explicitly reported the numeric tokenization strategy of the models they evaluated.

Limitations and Future Work

  • Scale and Scope: The controlled experiments were limited to small models trained from scratch on 3-digit addition. While the underlying mechanism is scale-independent, the exact magnitude of this effect in frontier LLMs remains unquantified.
  • Standard Autoregressive Training Only: The study focuses strictly on standard next-token prediction. It does not explore how output supervision interacts with advanced post-training methods like Reinforcement Learning from Human Feedback (RLHF), process supervision, or Chain-of-Thought (CoT) prompting.
  • Probing Constraints: Probing measures geometric decodability rather than the absolute presence or absence of information.
  • Future Directions: The authors suggest extending this output supervision framework to other structured domains where tokenization choices vary, such as morphology, date expressions in temporal reasoning, and code generation.

Summary updated

Abstract

Tokenization in language models is treated by default as an input preprocessing decision. We argue that this framing is incomplete: in autoregressive models, tokenizer granularity determines what the model must resolve in a single forward pass, and therefore the supervision signal it receives. This affects both the difficulty of the learning problem and the representations that emerge inside the model. We test this in a controlled experiment on numeric reasoning with a novel decoupling of input and output tokenization. As the output supervision view predicts, differences in task performance, training dynamics, and model internals are induced by output tokenization and largely invariant to input tokenization. This may matter in practice, because models with different tokenization strategies differ not only in input representation but in the task they were trained on. Comparisons between models may thus partly reflect task definition rather than ability. A survey of 120 recent *CL papers on numeric reasoning confirms that this is rarely acknowledged: only about 10% report the numeric tokenization of the models they evaluate, while 69% compare across tokenization, and thus supervision, regimes without reporting it. While prior work documents that tokenization consistently affects model performance, there is no principled account of why. We argue that framing tokenization as output supervision provides that account.
BibTeX
@misc{baeumel2026when,
  author = {Tanja Baeumel and Josef van Genabith and Simon Ostermann},
  title = {{When Tokenization is Secretly Output Supervision}},
  year = {2026},
  eprint = {2609.01386},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.01386},
}
Figure