More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

Xinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He, Fabian Waschkowski, Lukas Wesemann, Peter H. Tu, Jing Zhang

Abstract

Reasoning has emerged as a pivotal capability in Large Language Models (LLMs). Through Reinforcement Learning (RL), typically Group Relative Policy Optimization (GRPO), these models are able to solve complex tasks such as mathematics and code generation. Building on these advances, recent research has sought to extend reasoning to Vision-Language Models (VLMs), yielding promising results across diverse visual tasks. Despite this progress, our study uncovers the dual nature of multimodal reasoning: while it substantially enhances logical inference and facilitates performance on challenging problems, it may gradually impair perceptual grounding, leading to recognition failures on otherwise basic visual questions. Through further analysis, we attribute this phenomenon to visual forgetting, wherein prolonged reasoning causes the model to increasingly disregard visual input. To address this, we propose Vision-Anchored Policy Optimization (VAPO), a simple yet effective method that explicitly steers the reasoning process toward visually grounded trajectories. Our result model, VAPO-Thinker-7B, significantly strengthens the model's reliance on visual information and achieves new state-of-the-art results on a wide range of established benchmarks.

Introduction: The "Visual Forgetting" Problem

In the world of Large Language Models (LLMs), "reasoning" (often implemented via Chain-of-Thought) has been a game-changer for solving complex mathematical and coding tasks. Naturally, researchers have sought to port these capabilities to Vision-Language Models (VLMs). However, this transition has revealed a counter-intuitive "No-Free-Lunch" dilemma: as a VLM spends more time "thinking" through a problem, its accuracy on basic visual details often decreases.

A Vision-Language Model fails a basic perception task (summing profits from a chart) despite a long reasoning chain, illustrating the 'visual forgetting' phenomenon.

This phenomenon, termed Visual Forgetting, occurs when a model becomes so engrossed in its internal reasoning monologue that it stops paying attention to the actual image tokens. This leads to hallucinations or recognition failures on visual elements that were perfectly clear at the start of the reasoning process.

The Dual Nature of Multimodal Reasoning

Empirical analysis reveals that multimodal reasoning is a double-edged sword. While it enables the handling of complex logic, it introduces specific vulnerabilities:

  • The Reasoning Peak: For many visual tasks, model accuracy actually peaks during the reasoning process and then falls. "Early decision"—stopping the model before it finishes its full thought—often yields better results than letting it complete a long reasoning chain.
  • Perception as the Bottleneck: Quantitative analysis shows that over 50% of reasoning failures in state-of-the-art VLMs are due to basic perception errors (such as miscounting or misreading labels) rather than logical flaws.
  • Attention Drift: As reasoning progresses, the model's decision-making becomes increasingly driven by its own previous linguistic output rather than the visual input tokens.
Empirical analysis of (A) accuracy trends during reasoning, (B) the distribution of error types, and (C) the impact of recovery methods on perception errors.

Vision-Anchored Policy Optimization (VAPO)

To mitigate visual forgetting, we introduce Vision-Anchored Policy Optimization (VAPO). This training method uses visual checkpoints to keep the model’s "eyes" on the image throughout the reasoning process:

  1. Visual Claim Generation: A high-end model generates "primitive visual claims" about an image (e.g., "The bus is green").
  2. Visual Anchors: During Reinforcement Learning (using the GRPO algorithm), the model's reasoning is truncated at random points to insert these claims as "anchors."
  3. Perception Reward: The model is rewarded not just for the final answer, but for correctly verifying these visual anchors throughout its thought process.
  4. Late-Emphasis Weighting: Training puts more weight on perceptual accuracy toward the end of the reasoning chain, where forgetting is most likely to occur.

Key Results

The resulting model, VAPO-Thinker-7B, was evaluated across ten benchmarks, including vision-intensive sets like MMStar and HallusionBench. It achieved a new state-of-the-art for 7B-scale models:

  • Overall Performance: Achieved average gains of 2–4% over strong baselines like Vision-R1.
  • Vision-Indispensable Tasks: On the MMStar benchmark, VAPO reached 63.0% accuracy, surpassing previous bests.
  • Mathematical Reasoning: On MathVista, it reached 75.6%, outperforming other open-source 7B models.
  • Robustness: Unlike baseline models whose accuracy plummeted during long reasoning chains, VAPO-Thinker maintained a steady upward trend in accuracy as it "thought" longer.

Conclusion

By explicitly rewarding visual grounding during the reasoning process, VAPO successfully addresses the visual forgetting dilemma. This approach ensures that as models become more capable of complex logic, they do not lose their fundamental ability to perceive the world accurately. Future work will explore extending these anchors to multi-image and video-based reasoning tasks.

Summary updated

Abstract

Reasoning has emerged as a pivotal capability in Large Language Models (LLMs). Through Reinforcement Learning (RL), typically Group Relative Policy Optimization (GRPO), these models are able to solve complex tasks such as mathematics and code generation. Building on these advances, recent research has sought to extend reasoning to Vision-Language Models (VLMs), yielding promising results across diverse visual tasks. Despite this progress, our study uncovers the dual nature of multimodal reasoning: while it substantially enhances logical inference and facilitates performance on challenging problems, it may gradually impair perceptual grounding, leading to recognition failures on otherwise basic visual questions. Through further analysis, we attribute this phenomenon to visual forgetting, wherein prolonged reasoning causes the model to increasingly disregard visual input. To address this, we propose Vision-Anchored Policy Optimization (VAPO), a simple yet effective method that explicitly steers the reasoning process toward visually grounded trajectories. Our result model, VAPO-Thinker-7B, significantly strengthens the model's reliance on visual information and achieves new state-of-the-art results on a wide range of established benchmarks. Project page: https://xytian1008.github.io/VAPO/
BibTeX
@inproceedings{tian2026more,
  author = {Xinyu Tian and Shu Zou and Zhaoyuan Yang and Mengqi He and Fabian Waschkowski and Lukas Wesemann and Peter H. Tu and Jing Zhang},
  title = {{More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=XpL5eqjCjF},
}
Figure