TL;DR
GuirlVG introduces a highly data-efficient reinforcement learning framework for Graphical User Interface Visual Grounding (GUI-VG) that shatters the reliance on massive supervised datasets. By systematically optimizing Group Relative Policy Optimization (GRPO) with a novel Soft Reward Function and an Adversarial KL Factor, GuirlVG achieves state-of-the-art performance using just 5.2K training samples—outperforming traditional supervised methods trained on over 10 million samples.
What Problem Does This Paper Address?
For GUI agents to interact with digital environments, they must excel at GUI Visual Grounding (GUI-VG)—the ability to locate actionable elements (like buttons or icons) on a screenshot based on textual instructions. Historically, achieving high accuracy required Supervised Fine-Tuning (SFT) on millions of meticulously annotated screenshots, incurring massive data curation and training costs.
As Multimodal Large Language Models (MLLMs) advance, they inherently absorb GUI knowledge during pretraining, making exhaustive post-training SFT increasingly redundant and inefficient. While rule-based reinforcement fine-tuning (RFT) has emerged as a cheaper alternative in reasoning domains, a naive application of RFT to GUI tasks actually underperforms basic SFT. This paper tackles a fundamental question: How do we optimally formulate reinforcement learning objectives to unlock data-efficient GUI visual grounding?
Methods
The authors introduce GuirlVG, built on top of the GRPO algorithm. Instead of accepting standard RFT defaults, they systematically re-engineered its core components:
- Soft Reward Function (SRF): Standard RFT strictly penalizes formatting errors (e.g., missing a JSON bracket or a closing tag), which introduces training instability. SRF relaxes these constraints by offering partial credit for tag presence and ignoring minor stylistic deviations, focusing purely on the extracted coordinates.
- Functional Goal Alignment: Rather than forcing the model to predict bounding boxes (Bbox) and calculating Intersection-over-Union (IoU), GuirlVG switches to direct point prediction paired with a binary In-Bbox reward (checking if the point falls inside the ground-truth box).
- Adversarial KL Factor: To prevent the model from over-optimizing or "reward hacking" high-value responses, the authors introduce a dynamic KL penalty modifier ($\alpha_i$). Instead of a static penalty, $\alpha_i$ scales proportionally to the achieved reward, adaptively strengthening regularization as rewards increase.
- Strategic Prompting: The framework strategically withholds image resolution information during training to force the model to learn robust spatial reasoning, only providing it at test time to refine final predictions.
Datasets
- Training: 5,200 samples randomly selected from ShowUI (a dataset containing visually rich website data and desktop data from OmniAct augmented via GPT-4o).
- Evaluation: Evaluated across three major GUI-VG benchmarks:
- ScreenSpot: Evaluates grounding across mobile, desktop, and web environments.
- ScreenSpot v2: An updated version of ScreenSpot with corrected annotation errors.
- ScreenSpot-Pro: A challenging benchmark focusing on high-resolution professional applications spanning 23 applications and 5 industries.
Key Insights and Ideas
- "SFT memorizes, RL generalizes": GuirlVG achieved massive gains on out-of-domain tasks. For instance, it outperformed baselines on mobile screenshots by +11.8% despite containing zero mobile-specific samples in its 5.2K training set.
- Rigid formatting rewards hurt RL training: Giving partial credit for formatting allows the model to focus its exploration on spatial reasoning rather than getting bogged down by strict JSON syntax.
- Task-aligned objectives matter: Aligning the reward directly with the evaluation metric (point-in-box accuracy rather than bounding box IoU) yields substantial performance boosts.
- Withholding context can foster better learning: Counterintuitively, hiding image resolution during training forces the model to build stronger internal spatial awareness, making the resolution cue highly effective when reintroduced at inference.
Key Results
GuirlVG demonstrates unprecedented data efficiency, achieving state-of-the-art results across all evaluated benchmarks:
- ScreenSpot: Reached 88.7% accuracy with 5.2K samples, outperforming OS-Atlas (trained on 13.58M samples) by +7.7%.
- ScreenSpot v2: Achieved a new state-of-the-art 91.9% average accuracy.
- ScreenSpot-Pro: Delivered an absolute gain of +17.2% over the best SFT baseline, proving its robust generalization capabilities in complex, high-resolution professional software environments.
Limitations and Future Work
The authors note a few limitations in their current study:
- Model Scope: The empirical exploration was restricted to a single MLLM architecture (Qwen2.5-VL), leaving its generalizability across other architectures unverified.
- Compute & Scale Constraints: Due to hardware limitations, the authors could not experiment with larger model scales (such as 32B or 72B parameter variants), which could yield further insights into how reinforcement learning scales for GUI tasks.
Summary updated