Latent Computation Tokens for Transformer Reasoning

50 concepts35 papers1 proposal

Standard autoregressive Transformers operate under a rigid computational constraint: they allocate exactly the same amount of processing power to every token they generate. Whether a model is outputting a simple punctuation mark or the crucial breakthrough in a complex mathematical proof, its internal representations undergo the exact same number of layer-by-layer transformations.

When faced with difficult reasoning tasks, this fixed-budget architecture creates a bottleneck. To solve a multi-step problem, a model must either compute the entire solution instantly in a single forward pass—which is often mathematically impossible for shallow networks—or externalize its thinking step-by-step using human-readable text, a method known as Chain-of-Thought (CoT) prompting.

As formalised by Nowak et al. in On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning, generating intermediate tokens effectively transforms a language model from a simple classifier into a sequential computer equipped with a working tape. However, relying entirely on text generation introduces significant latency, token costs, and a dependency on human-readable logic.

This realization has driven a new research direction: latent computation tokens. Instead of forcing a model to speak its thoughts aloud, researchers are injecting non-semantic placeholder tokens, filler characters, or continuous latent states directly into the sequence. These tokens act as an internal, high-dimensional scratchpad, giving the model the extra "time to think" silently before committing to an output.


The Core Mechanism: Expanding the Computational Graph

To understand why latent computation tokens work, we must look at how a Transformer processes a sequence. In a standard setup, each token position in the input sequence corresponds to a vertical column of hidden states across $L$ layers. Information flows upward through self-attention and feed-forward networks.

When we insert non-informative "pause tokens" or filler symbols (such as repeated periods or spaces) into the sequence before the final answer prompt, we are horizontally expanding the model's computational graph.

Standard Transformer (Fixed Compute):
[Prompt] -> [Layer 1] -> [Layer 2] -> [Layer L] -> [Immediate Answer]

With Latent Computation Tokens (Expanded Workspace):
[Prompt] -> [Pause Token 1] -> [Pause Token 2] -> [Pause Token 3] -> [Final Answer]
               │                 │                 │
               └─(Attention)─────┴─(Attention)─────┴─(Internal State Refinement)

During the generation of these meaningless tokens, the model's attention mechanism remains fully active. As Jang et al. demonstrate in Expanding Computation Spaces of LLMs at Inference Time, the hidden states of these filler tokens selectively attend back to key segments of the input prompt. Rather than wasting the forward pass, the network uses these extra cycles to perform relevant internal computation, information retrieval, and state transformations before generating a meaningful response.


The High-Dimensional Scratchpad

What exactly happens inside these latent steps? Instead of writing out text, the model uses the hidden vectors of the placeholder tokens as a temporary memory buffer and processing workspace.

In a comprehensive survey of this paradigm, Li et al. in Implicit Reasoning in Large Language Models: A Comprehensive Survey characterize these non-interpretable latent tokens as a high-dimensional computational workspace. This workspace allows the model to route information and resolve complex global dependencies that cannot be computed in a single pass.

Several empirical studies highlight what the model stores in this hidden space:


Decoupling Compute from Semantics

A highly surprising and intuitive finding in this space is that the semantic meaning of intermediate tokens is often secondary to the computational budget they provide.

In Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens, Valmeekam et al. trained models on complex tasks using intermediate reasoning chains that were entirely stripped of logical meaning or relevance. Crucially, these "reasonless" tokens still provided a massive performance boost. This confirms that the benefit of intermediate steps stems fundamentally from providing an expanded computational budget rather than the literal text generated.

This expansion is not just an optimization trick; it alters the theoretical limits of the architecture. London et al. provided a formal proof in Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers, showing that pause tokens strictly increase the expressivity of constant-depth Transformers. By distributing complex computations across the sequence length, the tokens increase the effective width of the computation, enabling shallow architectures to perform operations that would otherwise require a much deeper model.


Implementation Paradigms

The community has explored several distinct ways to implement this internal thinking space, ranging from training-free inference tricks to architectural overhauls.

1. Discrete Token Insertion

The simplest approach involves inserting non-semantic text characters (like periods or spaces) or dedicated learnable tokens (like <pause>) directly into the token sequence. While some inference-time gains are possible (Jang et al.), Goyal et al. establish in Think before you speak: Training Language Models With Pause Tokens that to fully exploit these extra cycles, a model must be exposed to these delays during pretraining. This ensures the model learns to navigate and utilize the expanded computational pathways.

2. Continuous Latent State Refinement

Rather than operating in the discrete space of tokens, some approaches insert intermediate hidden states directly into the continuous vector space between the input and the predicted output token. For instance, Zeng et al. introduce this in PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space, allowing the model to refine its internal representations in a continuous vector space before committing to a discrete token prediction. Similarly, Ji et al. utilize a continuous "Chain-of-Deliberation" in Learning Refined Document Representations for Dense Retrieval via Deliberate Thinking to generate deeply nuanced document embeddings without the latency of text generation.

3. Recurrent and Dynamic Depth Processing

Instead of expanding the sequence horizontally, some frameworks scale computation vertically by passing a single token's representation through recurrent loops or dynamic internal pathways. Zhang et al. propose this in MoDr: Mixture-of-Depth-Recurrent Transformers for Test-Time Reasoning, using a multi-cycle recurrent reasoning phase where a router dynamically selects specialized internal experts for each processing cycle. This allows the model to scale its internal compute dynamically based on the difficulty of the problem.

By shifting the computational burden from externalized, human-readable text to internal, high-dimensional latent spaces, latent computation tokens offer a path toward models that are faster, more expressive, and natively aligned with the continuous processing capabilities of neural networks.

Key Foundational Work

Exploring Automatically Perturbed Natural Language Explanations in Relation Extraction (2023) — contributed 1 concept elements to this direction.
Analysing Mathematical Reasoning Abilities of Neural Models (2019) — contributed 1 concept elements to this direction.
Replacing Linguists with Dummies: A Serious Need for Trivial Baselines in Multi-Task Neural Machine Translation (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_computation_tokens_47ef1001,
  title = {latent computation tokens},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-computation-tokens/rdr_47ef10012943433d9d6d65f971279f53},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_computation_tokens_47ef1001_v1,
  title = {latent computation tokens},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-computation-tokens/rdr_47ef10012943433d9d6d65f971279f53/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake