Tuning Qwen2.5-VL to Improve Its Web Interaction Skills

Alexandra Yakovleva, Henrik Pärssinen, Harri Valpola, Juho Kannala, Alexander Ilin

TL;DR

This paper introduces a two-stage self-distillation fine-tuning method that significantly improves the web interaction capabilities of the open-source vision-language model Qwen2.5-VL-32B. By training the model to accurately track a visually rendered cursor and ignore history-induced confirmation biases, the authors transition the agent from open-loop execution to closed-loop self-correction. This approach boosts single-click task success rates from 86% to 94% in the most challenging evaluation settings.

What Problem Does This Paper Address?

Most current web-navigating AI agents rely heavily on text-based representations like HTML source code, DOM trees, or accessibility trees. While efficient, text-only agents fail when the underlying code doesn't accurately reflect the visual layout, or when critical information is embedded inside images, charts, or complex styling.

Purely visual web agents—which operate solely on screenshots—offer a more robust, human-like alternative. However, the authors identify three major bottlenecks that prevent state-of-the-art vision-language models (VLMs) from succeeding in this vision-only setup:

  1. Inaccurate Spatial Grounding: Models struggle to precisely localize the cursor, the target element, and the spatial relationship between them.
  2. Prompt Sensitivity: Small changes in instruction phrasing cause erratic action selections.
  3. Overoptimistic History Bias: When models look at their own past action history (the interaction trace), they tend to blindly assume their previous actions succeeded perfectly. This leads to "open-loop" behavior, where the model combines multiple actions (e.g., move + click) without visually verifying if the cursor actually reached the target first.

Methods

To address these challenges, the authors restrict the agent's action space to two atomic operations—mouse_move(x, y) and mouse_click()—and build a custom simulation environment called VisionWebBrowser using Playwright. Crucially, this environment explicitly renders a macOS-style cursor directly onto the webpage screenshots to serve as a visual anchor for the model.

The core contribution is a Two-Stage Fine-Tuning Pipeline utilizing self-distillation, where a student model learns from a teacher model (both based on Qwen2.5-VL-32B):

Overview of the two-stage fine-tuning pipeline.
  • Stage 1: Learning to Act from the Screen. This stage focuses on spatial grounding. The teacher model is given privileged "hints" (ground-truth target coordinates and the correct action). The student model learns to look at the screenshot and task description to accurately predict whether it needs to move or click, and outputs the exact coordinates if a move is required.
  • Stage 2: Focusing on the Screen State. This stage breaks the overoptimistic history bias. The authors feed the student model a distracting interaction history trace, but train it to match a teacher that generates unbiased reasoning based only on the current screen state. This teaches the student to prioritize visual reality over its past intentions, while also internalizing behavioral guidelines so that explicit prompting is no longer required.

Datasets

The authors evaluated their approach on a newly introduced benchmark:

  • single-click_bench: A custom dataset consisting of 430 single-click web tasks mapped to static HTML snapshots to ensure evaluation reproducibility.

Tasks are evaluated under two text formulations:

  • Simplified: Automatically generated templates (e.g., "Click on the element that displays {text}").
  • Human-like: Natural user queries generated by Qwen2.5-VL-32B to emulate real-world variability (e.g., "Switch the website's language to English").

Key Insights and Ideas

  • The Cursor as a Causal Anchor: Explicitly rendering the cursor on the screenshot is vital. It allows a single-module VLM to perceive its own state relative to the environment and build a causal understanding of how its actions alter the screen.
  • History is a Double-Edged Sword: While interaction history is vital for complex multi-step planning, it introduces a powerful confirmation bias in VLMs. Models frequently trust their recalled intent ("I planned to move to X") over visual evidence ("The cursor is actually at Y").
  • Emergence of Closed-Loop Control: By explicitly training the model to overcome history-induced noise in Stage 2, the agent naturally develops a "closed-loop" habit. If an initial mouse movement misses the target, the model can re-evaluate the new screen state and adjust its coordinates rather than blindly proceeding to click.

Key Results

  • Action Selection: Stage 1 fine-tuning drastically improved the model's ability to choose between moving and clicking, raising validation accuracy from 0.79 to 0.94, approaching the privileged teacher's score of 0.99.
Table 1: Move vs. click selection accuracy.
  • Downstream Success Rate: On the full benchmark with the interaction trace visible (the most distracting setting), the baseline Qwen2.5-VL model's success rate dropped to 86-87%. The Stage 2 fine-tuned model successfully mitigated this performance degradation, achieving a 94-95% success rate even without explicit behavioral prompts.
Table 2: Accuracy on single-click tasks.
  • Closed-Loop Correction ($R_{corr}$): The baseline and Stage 1 models achieved an $R_{corr}$ of nearly 0%, meaning they almost never corrected a missed mouse movement. The Stage 2 model achieved a 35% correction rate, proving it successfully learned to self-correct based on visual feedback.

Limitations and Future Work

While promising, the scope of this work is intentionally constrained:

  • Simplified Action Space: The environment is currently restricted to single-click tasks using only move and click commands. It does not yet incorporate scrolling, typing, or dragging.
  • Static Pages: The benchmark relies on static HTML snapshots. Although the authors filter out tasks where dynamic elements (like cookie banners) obscure the target, real-world web environments are highly dynamic and prone to layout shifts.

Future Directions: The authors plan to scale this two-stage fine-tuning framework to broader action spaces and evaluate its closed-loop control capabilities on complex, multi-step web interactions.

Summary updated

Abstract

Recent advances in vision-language models (VLMs) have sparked growing interest in using them to automate web tasks, yet their feasibility as independent agents that reason and act purely from visual input remains underexplored. We investigate this setting using Qwen2.5-VL-32B, one of the strongest open-source VLMs available, and focus on improving its reliability in web-based control. Through initial experimentation, we observe three key challenges: (i) inaccurate localization of target elements, the cursor, and their relative positions, (ii) sensitivity to instruction phrasing, and (iii) an overoptimistic bias toward its own actions, often assuming they succeed rather than analyzing their actual outcomes. To address these issues, we fine-tune Qwen2.5-VL-32B for a basic web interaction task: moving the mouse and clicking on a page element described in natural language. Our training pipeline consists of two stages: (1) teaching the model to determine whether the cursor already hovers over the target element or whether movement is required, and (2) training it to execute a single command (a mouse move or a mouse click) at a time, verifying the resulting state of the environment before planning the next action. Evaluated on a custom benchmark of single-click web tasks, our approach increases success rates from 86% to 94% under the most challenging setting.
BibTeX
@inproceedings{yakovleva2026tuning,
  author = {Alexandra Yakovleva and Henrik Pärssinen and Harri Valpola and Juho Kannala and Alexander Ilin},
  title = {{Tuning Qwen2.5-VL to Improve Its Web Interaction Skills}},
  booktitle = {The Web Conference},
  year = {2026},
  doi = {10.1145/3774904.3792932},
}
Figure