More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

Xinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He, Fabian Waschkowski, Lukas Wesemann, Peter Tu, Jing Zhang

TL;DR

While long-form reasoning (like Chain-of-Thought) significantly boosts the logical capabilities of Vision-Language Models (VLMs), this paper reveals a striking downside: longer reasoning paths cause models to suffer from "visual forgetting," leading to basic perceptual errors. To counteract this, the authors introduce Vision-Anchored Policy Optimization (VAPO), a reinforcement learning technique that forces models to maintain visual grounding throughout their entire thinking process, setting a new state-of-the-art across multiple multimodal benchmarks.

What Problem Does This Paper Address?

Recent breakthroughs in Large Language Models (LLMs) have shown that giving a model time to "think"—generating step-by-step reasoning before answering—massively improves performance on complex tasks like math and coding. Naturally, researchers have begun applying this test-time scaling paradigm to Vision-Language Models (VLMs) using Reinforcement Learning (RL) algorithms like Group Relative Policy Optimization (GRPO).

However, the authors of this paper uncover a counter-intuitive bottleneck unique to the multimodal domain: the harder a VLM thinks, the worse it sees.

As a VLM generates a long string of reasoning tokens, its attention mechanism progressively shifts away from the initial visual tokens and focuses almost entirely on its own historical text. This phenomenon, termed visual forgetting, causes the model to hallucinate or misinterpret basic visual details in the later stages of reasoning, leading to catastrophic failures on straightforward visual questions.

Figure 1: The examples where more reasoning yields less accuracy. We select Vision-R1 (Huang et al., 2025) as a representitive multimodal reasoning model and evaluate on established benchmarks. Given the ground truth (GT) answer, we examine three settings, i.e., direct (DT) answering, full reasoning, and early decision. Correct and incorrect responses are highlighted in green and red, respectively, while designed prompts for early decision are indicated in blue.

Methods

To diagnose and fix visual forgetting, the authors first introduce an Early Decision Mode as an analytical tool. By forcing the model to halt its reasoning at intermediate logical boundaries (like commas or periods) and answer immediately, they map out an accuracy-vs-reasoning-length curve.

To fundamentally solve the issue during training, they propose Vision-Anchored Policy Optimization (VAPO), which modifies the standard GRPO framework:

  1. Visual Claims Generation: Using an LLM (GPT-5), they generate a balanced set of simple, independent, factually correct and incorrect claims about the image (e.g., "The gray bus is facing toward the left").
  2. Inserting Visual Anchors: During the RL rollout phase, VAPO randomly samples intermediate checkpoints along the model's reasoning trajectory. At each checkpoint, the model is probed with a visual claim and must output a simple binary judgment (yes/no).
  3. Perception Reward: The framework introduces a new reward based on the accuracy of these binary judgments. Crucially, they use a late-emphasis weighted aggregation ($\beta$), which assigns higher penalties to perceptual failures that occur later in the reasoning chain, precisely where visual forgetting is most severe.

An accuracy gate is also implemented to ensure the model doesn't "reward hack" by simply shortening its reasoning paths to maintain high perception scores.

Datasets

The authors trained their models on ViRL39K, a high-quality, curated dataset combining samples from MM-Eureka, MV-Math, and M3CoT.

To comprehensively evaluate their approach, they tested across ten diverse benchmarks:

  • Reasoning/Math-centric: MathVerse, MathVista, MathVision, LogicVista, WeMath, and Geometry3k.
  • Vision-intensive/Generalist: MMMU, MMStar, HallusionBench, and MMVet.
  • Multi-image tasks (Pilot Study): BLINK and MuirBench.

Key Insights and Ideas

  • The Dual Nature of Multimodal Reasoning: Long reasoning chains act as a double-edged sword. While they enhance symbolic and logical inference, they actively degrade perceptual grounding. Error analysis shows that over 50% of full-reasoning failures are basic perception errors, many of which are completely rectifiable if the model is forced to answer early.
  • The Mechanism of Visual Forgetting: Tracking attention ratios revealed a steep, monotonic decline in attention allocated to visual tokens as generation steps progressed. The model essentially blinds itself with its own internal monologue.
  • Inference Remedies vs. Training Fixes: Simple test-time fixes like visual replay (re-inserting the image periodically) or focus prompts (instructing the model to "look back") help mitigate the drop, but they introduce heavy computational overhead. VAPO solves this during training, requiring only a single-token (yes/no) output during the anchor-checking phase, adding a negligible 30 minutes to a standard 19-hour training pipeline.
Analysis showing how accuracy drops as reasoning progresses, alongside error distributions.

Key Results

The resulting model, VAPO-Thinker-7B (built on Qwen2.5-VL-7B), achieved dominant performance across the board:

  • General Multimodal Benchmarks: Surpassed previous state-of-the-art models by an average of 3.2% on vision-intensive tasks (jumping from 59.9% to 63.1% overall), with particularly strong gains on MMStar and HallusionBench.
  • Mathematical Benchmarks: Achieved a steady 2.0% average improvement over strong baselines like Vision-R1-7B.
  • Perceptual Stability: While vanilla models saw their visual anchor verification scores drop to near-random guessing (~0.5) by the end of long reasoning paths, VAPO-Thinker maintained a stable, high perceptual accuracy of ~0.9 throughout the entire generation process.

Limitations and Future Work

  • Dependency on Claim Generators: The performance ceiling of VAPO is tightly bound to the quality and accuracy of the visual claims used as anchors. Relying on proprietary models like GPT-5 for data generation introduces potential noise or biases.
  • Hyperparameter Tuning: The framework introduces several sensitive hyperparameters, such as the anchor count ($K$) and late-emphasis weight ($\beta$). The authors note that logic-heavy tasks prefer different configurations than vision-heavy tasks, suggesting a need for adaptive, dynamic parameter adjustment in the future.
  • Expansion Beyond Single Images: The primary evaluation focused on single-image tasks. While a small pilot study on multi-image tasks (BLINK, MuirBench) yielded promising results, robustly scaling VAPO to video and complex sequential multi-image reasoning remains an open avenue for future research.

Summary updated

Abstract

Reasoning has emerged as a pivotal capability in Large Language Models (LLMs). Through Reinforcement Learning (RL), typically Group Relative Policy Optimization (GRPO), these models are able to solve complex tasks such as mathematics and code generation. Building on these advances, recent research has sought to extend reasoning to Vision-Language Models (VLMs), yielding promising results across diverse visual tasks. Despite this progress, our study uncovers the dual nature of multimodal reasoning: while it substantially enhances logical inference and facilitates performance on challenging problems, it may gradually impair perceptual grounding, leading to recognition failures on otherwise basic visual questions. Through further analysis, we attribute this phenomenon to visual forgetting, wherein prolonged reasoning causes the model to increasingly disregard visual input. To address this, we propose Vision-Anchored Policy Optimization (VAPO), a simple yet effective method that explicitly steers the reasoning process toward visually grounded trajectories. Our result model, VAPO-Thinker-7B, significantly strengthens the model's reliance on visual information and achieves new state-of-the-art results on a wide range of established benchmarks. Project page: https://xytian1008.github.io/VAPO/
BibTeX
@inproceedings{tian2026more,
  author = {Xinyu Tian and Shu Zou and Zhaoyuan Yang and Mengqi He and Fabian Waschkowski and Lukas Wesemann and Peter Tu and Jing Zhang},
  title = {{More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=XpL5eqjCjF},
}
Figure