TL;DR
This paper introduces a two-stage self-distillation fine-tuning method that significantly improves the web interaction capabilities of the open-source vision-language model Qwen2.5-VL-32B. By training the model to accurately track a visually rendered cursor and ignore history-induced confirmation biases, the authors transition the agent from open-loop execution to closed-loop self-correction. This approach boosts single-click task success rates from 86% to 94% in the most challenging evaluation settings.
What Problem Does This Paper Address?
Most current web-navigating AI agents rely heavily on text-based representations like HTML source code, DOM trees, or accessibility trees. While efficient, text-only agents fail when the underlying code doesn't accurately reflect the visual layout, or when critical information is embedded inside images, charts, or complex styling.
Purely visual web agents—which operate solely on screenshots—offer a more robust, human-like alternative. However, the authors identify three major bottlenecks that prevent state-of-the-art vision-language models (VLMs) from succeeding in this vision-only setup:
- Inaccurate Spatial Grounding: Models struggle to precisely localize the cursor, the target element, and the spatial relationship between them.
- Prompt Sensitivity: Small changes in instruction phrasing cause erratic action selections.
- Overoptimistic History Bias: When models look at their own past action history (the interaction trace), they tend to blindly assume their previous actions succeeded perfectly. This leads to "open-loop" behavior, where the model combines multiple actions (e.g., move + click) without visually verifying if the cursor actually reached the target first.
Methods
To address these challenges, the authors restrict the agent's action space to two atomic operations—mouse_move(x, y) and mouse_click()—and build a custom simulation environment called VisionWebBrowser using Playwright. Crucially, this environment explicitly renders a macOS-style cursor directly onto the webpage screenshots to serve as a visual anchor for the model.
The core contribution is a Two-Stage Fine-Tuning Pipeline utilizing self-distillation, where a student model learns from a teacher model (both based on Qwen2.5-VL-32B):
- Stage 1: Learning to Act from the Screen. This stage focuses on spatial grounding. The teacher model is given privileged "hints" (ground-truth target coordinates and the correct action). The student model learns to look at the screenshot and task description to accurately predict whether it needs to move or click, and outputs the exact coordinates if a move is required.
- Stage 2: Focusing on the Screen State. This stage breaks the overoptimistic history bias. The authors feed the student model a distracting interaction history trace, but train it to match a teacher that generates unbiased reasoning based only on the current screen state. This teaches the student to prioritize visual reality over its past intentions, while also internalizing behavioral guidelines so that explicit prompting is no longer required.
Datasets
The authors evaluated their approach on a newly introduced benchmark:
- single-click_bench: A custom dataset consisting of 430 single-click web tasks mapped to static HTML snapshots to ensure evaluation reproducibility.
Tasks are evaluated under two text formulations:
- Simplified: Automatically generated templates (e.g., "Click on the element that displays {text}").
- Human-like: Natural user queries generated by Qwen2.5-VL-32B to emulate real-world variability (e.g., "Switch the website's language to English").
Key Insights and Ideas
- The Cursor as a Causal Anchor: Explicitly rendering the cursor on the screenshot is vital. It allows a single-module VLM to perceive its own state relative to the environment and build a causal understanding of how its actions alter the screen.
- History is a Double-Edged Sword: While interaction history is vital for complex multi-step planning, it introduces a powerful confirmation bias in VLMs. Models frequently trust their recalled intent ("I planned to move to X") over visual evidence ("The cursor is actually at Y").
- Emergence of Closed-Loop Control: By explicitly training the model to overcome history-induced noise in Stage 2, the agent naturally develops a "closed-loop" habit. If an initial mouse movement misses the target, the model can re-evaluate the new screen state and adjust its coordinates rather than blindly proceeding to click.
Key Results
- Action Selection: Stage 1 fine-tuning drastically improved the model's ability to choose between moving and clicking, raising validation accuracy from 0.79 to 0.94, approaching the privileged teacher's score of 0.99.
- Downstream Success Rate: On the full benchmark with the interaction trace visible (the most distracting setting), the baseline Qwen2.5-VL model's success rate dropped to 86-87%. The Stage 2 fine-tuned model successfully mitigated this performance degradation, achieving a 94-95% success rate even without explicit behavioral prompts.
- Closed-Loop Correction ($R_{corr}$): The baseline and Stage 1 models achieved an $R_{corr}$ of nearly 0%, meaning they almost never corrected a missed mouse movement. The Stage 2 model achieved a 35% correction rate, proving it successfully learned to self-correct based on visual feedback.
Limitations and Future Work
While promising, the scope of this work is intentionally constrained:
- Simplified Action Space: The environment is currently restricted to single-click tasks using only move and click commands. It does not yet incorporate scrolling, typing, or dragging.
- Static Pages: The benchmark relies on static HTML snapshots. Although the authors filter out tasks where dynamic elements (like cookie banners) obscure the target, real-world web environments are highly dynamic and prone to layout shifts.
Future Directions: The authors plan to scale this two-stage fine-tuning framework to broader action spaces and evaluate its closed-loop control capabilities on complex, multi-step web interactions.
Summary updated