OmniParser for Pure Vision Based GUI Agent

Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah

Abstract

OmniParser is a comprehensive screen-parsing framework designed to bridge the gap between raw user interface (UI) screenshots and the reasoning capabilities of Vision Language Models (VLMs). By transforming visual data into structured, machine-readable maps, OmniParser enables agents to navigate diverse operating systems—including Windows, macOS, Android, and iOS—using pure vision. This approach addresses critical challenges in precision and semantic understanding, often outperforming models that rely on underlying metadata like HTML.

Introduction

Large Vision Language Models (VLMs) such as GPT-4V possess significant reasoning potential but frequently struggle with the practicalities of GUI interaction. Two primary obstacles hinder their effectiveness: a lack of precision in generating exact x-y coordinates for clicks and "semantic blindness," where the function of a visual element (e.g., a specific icon) is unclear without metadata. While previous solutions like Set-of-Mark (SoM) prompting have attempted to mitigate these issues, they typically require access to source code (DOM/HTML), limiting their utility in desktop or mobile environments where such data is inaccessible. OmniParser introduces a vision-only pipeline to make GUI agents truly platform-agnostic.

Methods

The OmniParser framework employs a multi-stage pipeline to parse screenshots into a format suitable for VLM consumption:

  1. Interactable Region Detection: A fine-tuned YOLOv8 model identifies interactable elements such as icons and buttons, specialized for the unique geometry of UI components.
  2. Local Semantics (Icon Captioning): A fine-tuned BLIP-2 model generates functional descriptions for detected icons (e.g., "settings button"), providing the VLM with necessary context.
  3. OCR Integration: Standard Optical Character Recognition (OCR) extracts text and its corresponding coordinates.
  4. Structured Prompting: The system overlays numeric IDs on the screenshot and provides a text-based list of element IDs, locations, and descriptions. This allows the VLM to interact via IDs rather than raw coordinates.
OmniParser transforms raw screenshots into structured data by detecting interactable regions and providing functional captions for icons across various platforms.

Datasets and Evaluation

The researchers developed two specialized datasets to train the framework's components:

  • Interactable Icon Detection Dataset: 67k screenshots with bounding boxes extracted from DOM trees for ground truth.
  • Icon Description Dataset: 7k icon-description pairs, using GPT-4o to generate functional labels for training the BLIP-2 captioner.

Evaluation was conducted across three major benchmarks: ScreenSpot (cross-platform), Mind2Web (web navigation), and AITW (Android navigation).

Key Results and Insights

OmniParser significantly enhanced GPT-4V's performance across all tested environments:

  • ScreenSpot: Accuracy on "Icon/Widget" tasks improved from 16.2% to 73.0%.
  • Mind2Web: The vision-only approach outperformed baselines with DOM access, suggesting that visual representation is often more reliable than messy underlying code.
  • AITW: Achieved a 57.7% overall score, surpassing specialized Android models.

Strikingly, the results indicate that "local semantics"—the functional description of an icon—are as vital as geometric precision. However, the authors noted a "context gap" where small icons viewed in isolation remain ambiguous, highlighting the need for models that balance local detail with global screen context.

Conclusion

OmniParser demonstrates that a pure vision-based approach is not only viable but often superior for GUI agents. While challenges remain regarding repeated identical elements and OCR granularity, the framework provides a robust foundation for platform-agnostic digital assistants. Future work will focus on training captioning models that are aware of the entire screenshot to further resolve functional ambiguities.

Summary updated

Abstract

The recent advancements of large vision language models shows their great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen. To fill these gaps, we introduce OmniParser, a comprehensive method for parsing general user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements. OmniParser significantly improves GPT-4V's performance on ScreenSpot benchmark. And on Mind2Web and AITW benchmark, OmniParser with screenshot only input outperforms the GPT-4V baselines requiring additional information outside of screenshot. We further demonstrate that OmniParser can seamlessly integrate with other vision language models, significantly enhancing their agentic capabilities.
BibTeX
@misc{lu2024omniparser,
  author = {Yadong Lu and Jianwei Yang and Yelong Shen and Ahmed Awadallah},
  title = {{OmniParser for Pure Vision Based GUI Agent}},
  year = {2024},
  eprint = {2408.00203},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.00203},
}
Figure