State over Tokens: Characterizing the Role of Reasoning Tokens

Mosh Levy, Zohar Elyoseph, Shauli Ravfogel, Yoav GoldbergarXiv 2025 · DOI · arXiv

Authors: Mosh Levy, Zohar Elyoseph, Shauli Ravfogel, and Yoav Goldberg

Introduction

In the current landscape of Large Language Models (LLMs), the phenomenon of "Chain-of-Thought" (CoT) prompting has become a cornerstone for improving performance on complex reasoning tasks. By generating intermediate steps before arriving at a final answer, models demonstrate significantly enhanced capabilities. However, this success has introduced a persistent paradox: the "faithfulness gap." Empirical studies frequently show that these reasoning traces—while effective—often omit critical logical steps, include irrelevant filler, or provide explanations that do not align with the model's actual internal processing.

This paper introduces the State over Tokens (SoT) framework to resolve this conceptual tension. We argue that the primary function of reasoning tokens is not to provide human-readable explanations, but to serve as externalized computational states. By reframing tokens as the sole persistent information carriers across stateless computation cycles, we can explain why reasoning traces are computationally essential yet often interpretatively unfaithful.

The Whiteboard Analogy

To understand the functional role of tokens, we propose a thought experiment called the "Whiteboard Analogy." This analogy shifts our perspective from viewing LLM output as a narrative to viewing it as a working memory.

The Whiteboard Analogy: A thought experiment illustrating the functional role of tokens as externalized state.

In this scenario, a solver with a 10-second memory reset must solve a complex problem. Their only way to maintain progress is to write a single word on a whiteboard before each reset. When they return, they read the board and add the next piece of information. Crucially, the whiteboard is not a "diary" for an outside observer; it is a functional tool for the solver. They might use shorthand, skip obvious steps, or write symbols that only make sense to their "future self." An outside observer might find the text incomplete or confusing, but for the solver, it is the necessary state required to reach the solution.

A Formal Framework for State

We formalize LLM generation as a recursive function $\mathcal{M}$ where the model operates in discrete cycles. In each cycle $k$, the model takes the current sequence of tokens (the state $S_k$) and produces a new token. This new token is then appended to the state to form $S_{k+1}$.

$$S_{k+1} = S_k \oplus \mathcal{M}(S_k)$$

Recursive state construction: Each generation cycle appends a new token to the existing state to enable future computation.

As shown in the recursive construction above, the tokens are the only persistent artifacts between cycles. Internal activations and "thoughts" vanish the moment a token is sampled. Therefore, the tokens must carry all the information necessary to dictate future computation. In the example of calculating a sequence (such as Catalan numbers), each token represents a milestone in the computation that allows the model to "remember" where it left off and what the next logical step should be.

Key Insights: Tokens vs. Text

The SoT framework leads to several critical insights regarding the nature of LLM reasoning:

  1. Ontological Divergence: Reasoning tokens inhabit two distinct roles simultaneously. They are a communicative medium for humans (text) and a functional state for the model (state). These roles are often in conflict. A token that is highly efficient for the model's computation might be "gibberish" to a human reader.
  2. Forward-Looking Nature: A state exists to enable future computation, not to describe past reasoning. This explains why models often skip steps they have already "internalized" in their weights; they only need to externalize what is required to generate the next token.
  3. The "Two Masters" Problem: When we fine-tune models to produce more "readable" or "faithful" explanations, we may inadvertently degrade their reasoning performance. By forcing the model to serve the "human master" (readability), we may be stripping away the information it needs to serve the "computational master" (state efficiency).

Redefining Interpretability

The SoT framework suggests a shift in the goals of mechanistic interpretability. Instead of asking "Is this a good explanation?", researchers should ask "How is the computational state encoded in these tokens?" The goal moves from reading the tokens as English to decoding them as a functional substrate.

This perspective also aids in trust calibration. Users should not treat a Chain-of-Thought as a literal transcript of the model's "mind," but rather as a series of computational stepping stones. Understanding that the model only externalizes what it needs to proceed—rather than everything it knows—is key to accurately assessing the reliability of LLM outputs.

Conclusion and Future Work

By characterizing reasoning tokens as state rather than text, we provide a functional explanation for the faithfulness gap. Future research should investigate whether natural language is the most efficient medium for this state-carrying role, or if "soft tokens" and non-linguistic vectors could provide more powerful reasoning capabilities. Furthermore, new interpretability methods are needed to track how information propagates through these token-states across multiple generation cycles, moving beyond the analysis of single forward passes.

Summary updated

Abstract

Large Language Models (LLMs) can generate reasoning tokens before their final answer to boost performance on complex tasks. While these sequences seem like human thought processes, empirical evidence reveals that they are not a faithful explanation of the model's actual reasoning process. To address this gap between appearance and function, we introduce the State over Tokens (SoT) conceptual framework. SoT reframes reasoning tokens not as a linguistic narrative, but as an externalized computational state -- the sole persistent information carrier across the model's stateless generation cycles. This explains how the tokens can drive correct reasoning without being a faithful explanation when read as text and surfaces previously overlooked research questions on these tokens. We argue that to truly understand the process that LLMs do, research must move beyond reading the reasoning tokens as text and focus on decoding them as state.
BibTeX
@misc{levy2025state,
  author = {Mosh Levy and Zohar Elyoseph and Shauli Ravfogel and Yoav Goldberg},
  title = {{State over Tokens: Characterizing the Role of Reasoning Tokens}},
  year = {2025},
  eprint = {2512.12777},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.12777},
}
Figure