Training Multi-Image Vision Agents via End2End Reinforcement Learning

Chengqi Dong, Chuhuai Yue, Hang He, Rongge Mao, Fenghe Tang, S. Kevin Zhou, Zekun Xu, Xiaohan Wang, Jiajun Chai, Wei Lin, Guojun YinarXiv 2025 · DOI · arXiv

Introduction

The evolution of Multimodal Large Language Models (MLLMs) has shifted toward creating vision agents capable of complex reasoning through interleaved text and visual tool use. However, existing open-source vision agents often struggle with multi-image scenarios and suffer from "visual attention decay"—a phenomenon where the model's focus on visual tokens diminishes as the reasoning chain lengthens. IMAgent addresses these challenges by employing a pure end-to-end reinforcement learning (RL) approach to train a multi-image vision agent equipped with specialized tools for visual confirmation and reflection.

The IMAgent Framework

IMAgent is built upon the Qwen2.5-VL-7B-Instruct base model and trained using the Generalized Proximal Policy Optimization (GRPO) algorithm. Unlike traditional methods that require expensive Supervised Fine-Tuning (SFT) to initialize tool-use capabilities, IMAgent utilizes a "Zero RL" paradigm.

The core of the agent's reasoning capability lies in its Multimodal Chain-of-Thought (MCoT) framework, which integrates two primary visual tools:

  • Visual Confirmation (zoom_in): Allows the agent to focus on specific regions of interest by generating bounding boxes for high-resolution cropping.
  • Visual Reflection (lookback_reuse): Enables the agent to re-insert previously seen images into the current context window, effectively forcing the model to re-examine visual evidence during complex multi-turn reasoning.
The Multimodal Chain-of-Thought (MCoT) framework featuring visual confirmation and reflection tools.

Overcoming Visual Attention Decay

A critical contribution of this work is the empirical analysis of attention dynamics in VLMs. The authors observe that during long-form reasoning, attention to image tokens typically flatlines, leading the model to hallucinate based on its own generated text. IMAgent's tools act as an "attention defibrillator," creating spikes in visual attention precisely when the model needs to verify details or synthesize information across multiple images.

Attention distribution dynamics showing how visual tools restore focus to image tokens during long reasoning chains.

Reinforcement Learning Strategy

To stabilize training in a Zero RL setting, the authors introduced two key mechanisms:

  1. Two-Level Mask Strategy: This includes an Action-Level Mask to ignore environment errors during early exploration and a Trajectory-Level Mask to discard entirely failed reasoning loops, preventing noisy gradients from destabilizing the policy.
  2. Reward Shaping: A "tool use gain" reward is implemented to incentivize the model to explore tool-based strategies. This prevents the model from converging on a sub-optimal policy that ignores tools to avoid early formatting penalties.

Experimental Results

The effectiveness of IMAgent was evaluated on the newly introduced MIFG-QA (Multi-Image Fine-Grained QA) dataset, which contains 10k complex multi-image pairs. IMAgent achieved a state-of-the-art score of 49.21%, significantly outperforming the base Qwen2.5-VL (41.98%) and specialized models like Pixel Reasoner (44.49%).

Furthermore, IMAgent maintains high performance on single-image benchmarks, scoring 88.48% on V Bench and 72.5%* on HR-Bench. These results demonstrate that the multi-image training and RL strategies do not compromise the model's general vision-language capabilities.

Conclusion

IMAgent represents a significant step forward in open-source vision agents. By bypassing SFT and utilizing targeted visual tools within an RL framework, the model learns emergent strategies for multi-image reasoning while successfully mitigating the problem of visual attention decay. This approach paves the way for more robust and autonomous multimodal agents capable of handling real-world visual complexity.

Summary updated

Abstract

Recent VLM-based agents aim to replicate OpenAI O3's "thinking with images" via tool use, yet most open-source methods restrict inputs to a single image, limiting their applicability to real-world multi-image QA tasks. To address this gap, we propose IMAgent, an open-source visual agent trained with end-to-end reinforcement learning for fine-grained single/multi-image reasoning. During inference, VLMs tend to gradually neglect visual inputs; to mitigate this issue, we design two dedicated tools for visual reflection and verification, enabling the model to actively refocus attention on image content. Beyond that, we, for the first time, reveal how tool usage enhances agent performance from an attention perspective. Equipped with a carefully designed two-layer motion trajectory masking strategy and tool-use reward gain, IMAgent acquires an effective tool-use paradigm through pure reinforcement learning, eliminating the need for costly supervised fine-tuning data. To further unleash the inherent tool-usage potential of the base VLM and fill data gaps, we construct a challenging, visually enriched multi-image QA dataset via multi-agent system. Extensive experiments validate that IMAgent achieves SOTA performance across mainstream single and multi-image benchmarks, and our in-depth analysis offers actionable insights for the community. Code and data will be released soon.
BibTeX
@misc{dong2025training,
  author = {Chengqi Dong and Chuhuai Yue and Hang He and Rongge Mao and Fenghe Tang and S. Kevin Zhou and Zekun Xu and Xiaohan Wang and Jiajun Chai and Wei Lin and Guojun Yin},
  title = {{Training Multi-Image Vision Agents via End2End Reinforcement Learning}},
  year = {2025},
  eprint = {2512.08980},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.08980},
}
Figure