Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding

Wanfu Wang, Qipeng Huang, Guangquan Xue, Xiaobo Liang, Juntao LiarXiv 2025 · DOI · arXiv

Abstract

Graphical User Interface (GUI) grounding is a critical task for AI agents, requiring the translation of natural language instructions into precise screen coordinates. Traditional models often fail due to visual noise and resolution loss in high-resolution screenshots. We present LASER (Learning Active Perception via Self-Evolving Preference Optimization), a framework that teaches Vision Language Models (VLMs) to "zoom in" on relevant UI elements before interacting. By combining Monte Carlo quality estimation with Direct Preference Optimization (DPO), LASER enables models to autonomously discover optimal visual search strategies. Our 7B-parameter model achieves state-of-the-art performance, outperforming models ten times its size on the ScreenSpot-Pro benchmark.

Introduction

Modern GUI grounding—the task of clicking the correct icon or text based on a user command—is notoriously difficult. Most current models follow a "single-step" paradigm, attempting to predict coordinates from a single look at a downsampled screenshot. This approach suffers from visual clutter and the loss of fine-grained details. Unlike humans, who scan a screen and then focus their gaze, standard models lack a mechanism for iterative visual refinement. LASER addresses this by eliciting "active perception," allowing models to refine their focus through intermediate cropping steps.

Methodology

The LASER framework utilizes a three-stage self-evolving pipeline to develop active perception without requiring expensive human-annotated zoom trajectories.

The three-stage LASER framework: SFT for active perception, DPO for region preference, and multi-step perception.
  1. Eliciting Active Perception (SFT): The model is first trained to predict a "focus region" (a crop) before the final click. Using a rejection sampling approach, we retain only trajectories where the model successfully reaches the target, creating an initial supervised fine-tuning (SFT) dataset.
  2. Learning Focused Region Preferences (DPO): To distinguish between helpful and distracting crops, we employ Monte Carlo Quality Estimation. By sampling multiple click attempts from a single crop, we identify "positive" regions that consistently lead to success. These pairs are used for Direct Preference Optimization (DPO), teaching the model which visual perspectives are most informative.
  3. Difficulty-Aware Multi-step Perception: For complex tasks, the model learns to iteratively zoom in (e.g., Screen → Toolbar → Specific Icon), adaptively deciding the number of steps based on UI complexity.

Key Insights and Results

Empirical analysis reveals a critical "Optimal Region" for GUI grounding. As shown in the performance curves, focusing on a region roughly 20-25% of the original image size yields the best results. Regions that are too small lose necessary contextual cues, while regions that are too large retain excessive background noise.

Performance vs. Focus Region Size, highlighting the optimal region for text and icon grounding.

LASER (built on a GTA1-7B backbone) achieved a score of 55.7 on the ScreenSpot-Pro benchmark, surpassing the much larger Qwen2.5-VL-72B (53.3). Compared to the base 7B model's accuracy of 26.8%, LASER provides a massive improvement of over 20 points. This demonstrates that "thinking with images"—applying Chain-of-Thought reasoning to visual crops—is a highly efficient way to scale model capability.

Conclusion

LASER introduces a robust method for self-evolving active perception in GUI agents. By discovering its own optimal visual search strategies, the model achieves superior grounding accuracy with high efficiency. Future work will focus on reducing the computational overhead of multi-step perception and integrating these capabilities into long-horizon autonomous agents.

Summary updated

Abstract

Vision Language Models (VLMs) have recently achieved significant progress in bridging visual perception and linguistic reasoning. Recently, OpenAI o3 model introduced a zoom-in search strategy that effectively elicits active perception capabilities in VLMs, improving downstream task performance. However, enabling VLMs to reason effectively over appropriate image regions remains a core challenge in GUI grounding, particularly under high-resolution inputs and complex multi-element visual interactions. In this work, we propose LASER, a self-evolving framework that progressively endows VLMs with multi-step perception capabilities, enabling precise coordinate prediction. Specifically, our approach integrate Monte Carlo quality estimation with Intersection-over-Union (IoU)-based region quality evaluation to jointly encourage both accuracy and diversity in constructing high-quality preference data. This combination explicitly guides the model to focus on instruction-relevant key regions while adaptively allocating reasoning steps based on task complexity. Comprehensive experiments on the ScreenSpot Pro and ScreenSpot-v2 benchmarks demonstrate consistent performance gains, validating the effectiveness of our method. Furthermore, when fine-tuned on GTA1-7B, LASER achieves a score of 55.7 on the ScreenSpot-Pro benchmark, establishing a new state-of-the-art (SoTA) among 7B-scale models.
BibTeX
@misc{wang2025learning,
  author = {Wanfu Wang and Qipeng Huang and Guangquan Xue and Xiaobo Liang and Juntao Li},
  title = {{Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding}},
  year = {2025},
  eprint = {2509.04243},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.04243},
}
Figure