Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?

Tianyi Zhang, Mahtab Bigverdi, Ranjay KrishnaarXiv 2026 · arXiv

TL;DR

While adding continuous "thought tokens" to Vision-Language Models (VLMs) yields clear accuracy gains, this paper reveals that models rarely use the actual content of these tokens for reasoning. By introducing the Token Replacement Test (TRT), the authors demonstrate that corrupting continuous thought tokens with random noise or zeros barely impacts performance, exposing a critical content-utility gap in modern "visual thinking" architectures.

What Problem Does This Paper Address?

To help Vision-Language Models handle complex spatial and geometric tasks, researchers have increasingly augmented them with continuous or latent non-textual tokens—essentially a "Chain-of-Thought" conducted in embedding space rather than natural language. While these auxiliary tokens consistently boost benchmark scores, it has remained a mystery why they help.

Does the model actually read the rich visual or geometric information encoded within these custom embeddings? Or do the gains stem from incidental confounds, such as giving the model more sequence length (token budget) to compute answers, providing a reliable positional anchor, or acting as a training-time regularizer? This paper introduces a rigorous diagnostic framework to separate true semantic token utilization from these alternative explanations.

Methods

The authors formalize a diagnostic principle called Ablate-to-Validate, which they instantiate as the Token Replacement Test (TRT). TRT intercepts the intermediate thought tokens during inference and replaces them with counterfactual alternatives while keeping the prompt, image, token budget, and decoding configurations completely identical.

Overview of the Token Replacement Test (TRT)

The TRT suite evaluates several replacement strategies to isolate specific confounds:

  • Identity re-injection: Verifies the interception pipeline works without modification.
  • Zero replacement: Replaces the thought span with zeros to test sensitivity to the total removal of content.
  • Random / Distribution-matched random: Replaces embeddings with random noise (or noise matched to the empirical token distribution) to test if the specific predicted content matters.
  • First-repeat: Replaces the entire thought span with the very first token, preserving the token budget and position while completely eliminating token diversity.
  • Oracle injection: Injects ground-truth visual data into the thought span to see if the model can exploit a perfect signal.

The authors evaluate this across a controlled depth-reasoning testbed using LLaVA-13B and Qwen2.5-VL-3B, comparing continuous thought tokens against a discrete token baseline (quantized via a VQ-VAE codebook). They also apply TRT to three popular off-the-shelf continuous visual-thinking systems: Mirage, Mull-Tokens, and CoVT.

Datasets

  • ADE20K: Used to train the depth-span models locally (19,279 image-depth pairs).
  • HardBLINK: A challenging relative depth reasoning benchmark derived from BLINK. It overlays multiple labeled points on an image and asks the model to determine which point is closest to the camera.
  • BLINK, VSP (Visual Spatial Planning), and CV-Bench: Used to evaluate the off-the-shelf visual-thinking models under the TRT framework.

Key Insights and Ideas

  • The Content-Utility Gap: For continuous thought tokens, the specific values inside the embeddings don't actually matter. Models perform almost identically whether they receive their own predicted thoughts, random noise, or a single repeated token.
  • Presence Over Content: Continuous thought tokens act more like a structural cue or a fixed computational scaffold rather than an information-rich bottleneck. The model profits from where the tokens are and how many there are, but ignores what they say.
  • Discrete Tokens Actually Work as Bottlenecks: This issue is highly representation-dependent. When models use discrete tokens (symbolic codebook indices), they show a strong dependence on content—randomizing discrete tokens destroys performance, and oracle tokens boost it significantly. Continuous spaces might simply be too high-dimensional, noisy, or redundant, allowing the model to exploit marginal statistics instead of true semantics.

Key Results

The Token Replacement Test reveals a stark contrast between continuous and discrete thought tokens:

  • Continuous Spans are Stubbornly Robust to Corruption: On the HardBLINK depth testbed, replacing a model's predicted continuous thought tokens with random noise or repeating the first token resulted in negligible performance drops. For example, LLaVA-13B achieved 74.46% accuracy with its own tokens and still maintained 72.58% accuracy when those tokens were replaced with pure random noise.
  • Oracle Continuous Tokens Provide No Headroom: Injecting perfect ground-truth depth embeddings barely changed performance (74.46% $\rightarrow$ 74.73% for LLaVA, and actually decreased slightly for Qwen2.5-VL).
Table 3. TRT for continuous depth spans. Replacements are ap- plied only to the injected depth vectors before the language back- bone consumes them.
  • Discrete Spans Behave Logically: In contrast, for the discrete token baseline on Qwen2.5-VL, random replacement caused a massive performance drop from 71.24% to 51.34%, while oracle injection boosted accuracy significantly to 80.64%.
  • Off-the-Shelf Models Show the Same Flaw: Testing Mirage, Mull-Tokens, and CoVT confirmed the trend. For instance, on CoVT, replacing the continuous "thinking" tokens with distribution-matched random noise achieved 74.65% accuracy compared to the original 76.26% baseline.

Limitations and Future Work

  • Testbed Focus: The controlled testbed focused heavily on relative depth reasoning. While the authors evaluated off-the-shelf models on broader benchmarks, developing similarly controlled testbeds for other modalities (like bounding boxes or segmentation masks) would strengthen these insights.
  • Lack of an Architectural Solution: While the paper successfully diagnoses the content-utility gap, it does not provide a definitive architectural fix to force models to utilize continuous thought content more effectively.

The authors conclude that reporting TRT-style diagnostics should become standard practice when proposing new latent-token mechanisms, ensuring future architectures build true information bottlenecks rather than accidental computational scaffolds.

Summary updated

Abstract

Vision-language models (VLMs) are increasingly augmented with continuous or latent non-textual tokens intended to support "visual thinking." Despite improved task accuracy, this alone does not show that models actually use these tokens for reasoning -- gains may arise from confounds such as added context length, special-token anchoring, or training-time regularization. We formalize a diagnostic principle, Ablate-to-Validate, for testing whether latent-token content is genuinely utilized, and instantiate it as the Token Replacement Test (TRT), a standardized suite of content-replacement ablations. TRT holds the prompt, image, token budget, and decoding fixed while replacing intermediate tokens with zero, random, first-repeat, or oracle alternatives, isolating whether performance depends on token content or merely on token presence. As a controlled testbed, we study relative depth reasoning with LLaVA-13B and Qwen2.5-VL-3B, training models to predict and consume continuous or discrete depth spans across multiple frozen encoders (SigLIP2, CLIP, DINOv2) and token budgets. We additionally apply TRT to three off-the-shelf visual-thinking systems (Mirage, Mull-Tokens, CoVT) on BLINK, VSP, and CV-Bench. Across all settings, accuracy gains are a misleading proxy for latent-token reasoning: VLMs retain most improvement even when token content is corrupted or replaced, revealing a persistent gap between having a latent channel and using it as an information bottleneck. We recommend TRT as a standard diagnostic alongside accuracy for any method introducing continuous thought tokens.
BibTeX
@misc{zhang2026ablate,
  author = {Tianyi Zhang and Mahtab Bigverdi and Ranjay Krishna},
  title = {{Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?}},
  year = {2026},
  eprint = {2605.21642},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.21642},
}
Figure