WinSpot: GUI Grounding Benchmark with Multimodal Large Language Models

Zheng Hui, Yinheng Li, Dan Zhao, Colby R. Banbury, Tianyi Chen, Kazuhito KoishidaACL 2025 · DOI

Introduction: The Challenge of Desktop Automation

While AI agents have made significant strides in navigating web and mobile interfaces, the Windows desktop environment remains a "wild west" for automation. On the web, agents can leverage structured metadata like HTML or Document Object Model (DOM) trees to identify and interact with elements. This structured data provides a clear map for AI to follow.

Web automation often relies on structured data like the DOM tree, which provides clear metadata for UI elements.

In contrast, Windows desktop applications are notoriously inconsistent. A button in a legacy Win32 application, a modern Universal Windows Platform (UWP) app, or a cross-platform Electron app may look identical to a human but lack a standardized underlying representation. Many desktop interfaces lack "accessibility trees," forcing AI agents to rely purely on visual perception—a process known as "GUI grounding."

The WinSpot Benchmark

To address this gap, researchers introduced WinSpot, a large-scale benchmark designed to evaluate and improve the ability of Multimodal Large Language Models (MLLMs) to interact with the Windows ecosystem. WinSpot provides over 5,000 human-validated instruction-to-coordinate pairs across 14 core application categories, including System Settings, File Management, and Productivity Tools. It is 21 times larger than previous Windows-specific benchmarks.

An example of GUI grounding in WinSpot: translating a natural language subtask into specific click coordinates on a Windows Settings screen.

Data Generation Framework

Creating a high-quality dataset of this scale required a sophisticated Two-Stage Labeling Framework:

  1. Harvesting & Filtering: Screenshots from over 700 popular Windows apps were collected via the Bing API and filtered for quality using the Phi-3-vision model.
  2. Automated Grounding: A proprietary ViT-BERT model detected interactable icons, while GPT-4o generated natural language instructions describing these elements.
  3. Human Verification: To ensure a "gold standard" dataset, human annotators manually verified 5,000 pairs, confirming that instructions accurately mapped to visual elements.

Key Insights: Why Desktop is Harder

The evaluation of various MLLMs on WinSpot revealed several critical insights into the current state of AI:

  • The "Web Bias": Models like GPT-4o perform reasonably well on web-based layouts (e.g., the Microsoft Store) because their training data is heavily weighted toward web content. However, they struggle significantly with "system-heavy" tasks like navigating the Windows Control Panel or File Explorer.
  • The Need for Visual-Only Grounding: Given the inconsistency of underlying code in Windows apps, the authors argue that future GUI agents must move toward "purely visual" grounding rather than relying on metadata.
  • Specialization vs. Generalization: Fine-tuned models like Uground vastly outperform generalist models like GPT-4o. This suggests that "spatial reasoning"—the ability to map language to precise coordinates—is a distinct skill that general LLMs have not yet mastered.

Performance Results

The researchers measured "click accuracy"—whether a model's predicted click falls within the correct bounding box. The results highlight a significant performance gap:

  • Uground led the field with 44.2% overall accuracy.
  • GPT-4o and GPT-4V managed only 16.5% and 18.3% accuracy, respectively.
  • Category Variance: Performance varied wildly by task. While Uground achieved 82.4% accuracy on "MS Store & Web" tasks, it plummeted to 27.2% for "File Management," underscoring the complexity of navigating hierarchical file systems compared to standard web layouts.

Limitations and Future Directions

While WinSpot is a major step forward, it currently focuses on static screenshots. Real-world automation involves dynamic elements like animations, pop-ups, and multi-step reasoning (e.g., "Find a file, right-click, and compress"). Future work will likely focus on "temporal dynamics"—enabling models to maintain context across a sequence of frames to perform complex, multi-stage operations.

By providing a rigorous testing ground, WinSpot sets the stage for a new generation of AI agents capable of truly understanding and navigating the diverse landscape of desktop computing.

Summary updated

BibTeX
@inproceedings{hui2025winspot,
  author = {Zheng Hui and Yinheng Li and Dan Zhao and Colby R. Banbury and Tianyi Chen and Kazuhito Koishida},
  title = {{WinSpot: GUI Grounding Benchmark with Multimodal Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.acl-short.85},
}
Figure