Introduction
The recent surge in "Deep Research" agents has showcased the potential for AI to autonomously navigate the web and synthesize information. However, these agents remain predominantly text-centric, often failing when tasks require interpreting visual data such as scientific diagrams, financial charts, or complex web layouts. To address this, researchers from Alibaba’s Tongyi Lab introduced WebWatcher, a multimodal deep research agent designed to bridge the gap between text-based search and visual reasoning.
The Challenge of Multimodal Research
Existing Vision-Language (VL) agents typically suffer from two primary limitations. First, many are "perception-only," capable of describing an image but unable to use that visual information as a pivot for multi-step investigations. Second, they often rely on rigid, template-based pipelines that lack the flexibility needed to navigate the "open-world" complexity of the live web. The core bottleneck identified by the authors is a lack of high-quality training data specifically designed for multi-step multimodal reasoning.
Methodology: Building a Visual Researcher
WebWatcher is built upon a three-pronged development strategy:
- QA-to-VQA Data Pipeline: To generate challenging training data, the team converted complex text-based QA pairs into Visual Question Answering (VQA) tasks. By "fuzzing" or masking key entities in questions (e.g., replacing a specific name with "the person in this image") and grounding them in authentic web images, the model is forced to use visual cues to initiate its research process.
- Cold-Start Supervised Fine-Tuning (SFT): The model was fine-tuned on 8,000 high-quality "ReAct" (Think-Act-Observe) trajectories. This teaches the agent the "grammar" of research—how to logically sequence tools such as OCR, web search, and code interpreters.
- Reinforcement Learning via GRPO: Using Group-Relative Policy Optimization (GRPO), the researchers refined the agent's decision-making. This stage optimizes tool orchestration, teaching the model when to continue searching and when it has gathered sufficient evidence.
WebWatcher operates with a comprehensive toolset, including Web Text and Image Search, a "Visit" tool for page summarization, a Code Interpreter for mathematical and data analysis, and an internal OCR engine.
Benchmarking and Results
The researchers introduced BrowseComp-VL, a new benchmark comprising 399 VQA pairs that require multi-step web navigation and cross-modal integration. Additionally, WebWatcher was evaluated against several high-difficulty benchmarks including Humanity’s Last Exam (HLE-VL), LiveVQA, and MMSearch.
As shown in the figure above, WebWatcher (specifically the 32B version based on Qwen2.5-VL) demonstrated state-of-the-art performance across all tested benchmarks. On the BrowseComp-VL benchmark, it achieved a score of 27.0%, doubling the performance of the GPT-4o baseline (13.4%). In the expert-level HLE-VL, it reached an average score of 18.2%, surpassing specialized reasoning models like o4-mini.
A notable finding is the "test-time scaling" effect. On the HLE benchmark, WebWatcher’s success rate increased from 13.6% (single attempt) to 41.9% when allowed 32 attempts (Pass@32). This suggests that systematic exploration and extended "thinking" time are crucial for solving highly complex problems.
Insights and Future Directions
The success of WebWatcher highlights the power of entity obfuscation in training data, which shifts the model's behavior from simple retrieval to active reasoning. Furthermore, Reinforcement Learning proved highly effective at teaching the agent the "cost" of tool use, leading to more precise and less redundant search behaviors.
Despite these gains, challenges remain. The multi-step reasoning process introduces higher inference latency compared to one-shot models. Additionally, agents can occasionally fall into reasoning loops or misinterpret tool outputs. Future work will focus on expanding the agent's capabilities to handle diverse file formats like complex PDFs and videos, while further optimizing the efficiency of the reasoning process.
Summary updated