Authors: Mosh Levy, Zohar Elyoseph, Shauli Ravfogel, and Yoav Goldberg
Introduction
In the current landscape of Large Language Models (LLMs), the phenomenon of "Chain-of-Thought" (CoT) prompting has become a cornerstone for improving performance on complex reasoning tasks. By generating intermediate steps before arriving at a final answer, models demonstrate significantly enhanced capabilities. However, this success has introduced a persistent paradox: the "faithfulness gap." Empirical studies frequently show that these reasoning traces—while effective—often omit critical logical steps, include irrelevant filler, or provide explanations that do not align with the model's actual internal processing.
This paper introduces the State over Tokens (SoT) framework to resolve this conceptual tension. We argue that the primary function of reasoning tokens is not to provide human-readable explanations, but to serve as externalized computational states. By reframing tokens as the sole persistent information carriers across stateless computation cycles, we can explain why reasoning traces are computationally essential yet often interpretatively unfaithful.
The Whiteboard Analogy
To understand the functional role of tokens, we propose a thought experiment called the "Whiteboard Analogy." This analogy shifts our perspective from viewing LLM output as a narrative to viewing it as a working memory.
In this scenario, a solver with a 10-second memory reset must solve a complex problem. Their only way to maintain progress is to write a single word on a whiteboard before each reset. When they return, they read the board and add the next piece of information. Crucially, the whiteboard is not a "diary" for an outside observer; it is a functional tool for the solver. They might use shorthand, skip obvious steps, or write symbols that only make sense to their "future self." An outside observer might find the text incomplete or confusing, but for the solver, it is the necessary state required to reach the solution.
A Formal Framework for State
We formalize LLM generation as a recursive function $\mathcal{M}$ where the model operates in discrete cycles. In each cycle $k$, the model takes the current sequence of tokens (the state $S_k$) and produces a new token. This new token is then appended to the state to form $S_{k+1}$.
$$S_{k+1} = S_k \oplus \mathcal{M}(S_k)$$
As shown in the recursive construction above, the tokens are the only persistent artifacts between cycles. Internal activations and "thoughts" vanish the moment a token is sampled. Therefore, the tokens must carry all the information necessary to dictate future computation. In the example of calculating a sequence (such as Catalan numbers), each token represents a milestone in the computation that allows the model to "remember" where it left off and what the next logical step should be.
Key Insights: Tokens vs. Text
The SoT framework leads to several critical insights regarding the nature of LLM reasoning:
- Ontological Divergence: Reasoning tokens inhabit two distinct roles simultaneously. They are a communicative medium for humans (text) and a functional state for the model (state). These roles are often in conflict. A token that is highly efficient for the model's computation might be "gibberish" to a human reader.
- Forward-Looking Nature: A state exists to enable future computation, not to describe past reasoning. This explains why models often skip steps they have already "internalized" in their weights; they only need to externalize what is required to generate the next token.
- The "Two Masters" Problem: When we fine-tune models to produce more "readable" or "faithful" explanations, we may inadvertently degrade their reasoning performance. By forcing the model to serve the "human master" (readability), we may be stripping away the information it needs to serve the "computational master" (state efficiency).
Redefining Interpretability
The SoT framework suggests a shift in the goals of mechanistic interpretability. Instead of asking "Is this a good explanation?", researchers should ask "How is the computational state encoded in these tokens?" The goal moves from reading the tokens as English to decoding them as a functional substrate.
This perspective also aids in trust calibration. Users should not treat a Chain-of-Thought as a literal transcript of the model's "mind," but rather as a series of computational stepping stones. Understanding that the model only externalizes what it needs to proceed—rather than everything it knows—is key to accurately assessing the reliability of LLM outputs.
Conclusion and Future Work
By characterizing reasoning tokens as state rather than text, we provide a functional explanation for the faithfulness gap. Future research should investigate whether natural language is the most efficient medium for this state-carrying role, or if "soft tokens" and non-linguistic vectors could provide more powerful reasoning capabilities. Furthermore, new interpretability methods are needed to track how information propagates through these token-states across multiple generation cycles, moving beyond the analysis of single forward passes.
Summary updated