TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents

Kunal Singh, Shreyas Singh, Mukund Khanna

TL;DR

This paper introduces TRISHUL, a novel framework that empowers Large Vision-Language Models (LVLMs) to understand graphical user interfaces (GUIs) more holistically. By combining hierarchical screen parsing and spatially enhanced element descriptions, TRISHUL significantly improves the ability of AI agents to interact with and comprehend GUIs, outperforming existing methods in both action grounding and GUI referring tasks.

What Problem Does This Paper Address?

Developing AI agents that can effectively interact with digital devices using natural language commands has been a long-standing goal. While Large Vision-Language Models (LVLMs) have shown promise, current approaches for building GUI agents face several limitations. Many rely on dataset-specific training, leading to poor generalization across different platforms and interfaces. Existing methods often focus on singular tasks like action grounding (mapping instructions to GUI elements) or GUI referring (describing elements), rather than achieving a comprehensive understanding of the entire GUI. Furthermore, some advanced LVLMs require structured metadata (like HTML) for action grounding, which is not always available or reliable. This paper aims to bridge this gap by proposing a training-free framework that enhances generalist LVLMs for robust and adaptable GUI comprehension.

Methods

TRISHUL is a training-free, agentic framework designed to equip LVLMs with advanced GUI understanding capabilities. It comprises two core modules:

  • Hierarchical Screen Parsing (HSP): This module organizes GUI elements into a multi-granular hierarchy. It distinguishes between "Global Regions of Interest" (GROIs), which are broader areas that group related components, and "Local Elements" (icons, text, buttons, images). This hierarchical structure helps capture spatial and semantic relationships within the GUI. The process involves using SAM (Segment Anything Model) for initial segmentation and EasyOCR for text detection, followed by filtering and non-max suppression to refine GROI candidates.
  • Spatially Enhanced Element Description (SEED): This module generates contextually aware and spatially informed functional descriptions for local GUI elements. It leverages Chain of Thought (CoT) and In-Context Learning (ICL) to pair nearby icons and text, creating high-fidelity descriptions that clarify each element's role. SEED classifies elements as "paired" (icon and text together) or "standalone" (relying on visual cues alone) and identifies "actionable" elements (like input fields or buttons).

These modules work synergistically to provide a rich, multi-layered representation of GUI screens, enhancing both action grounding and GUI referring capabilities.

Datasets

The paper evaluates TRISHUL on several established benchmarks:

  • ScreenSpot: A dataset for GUI navigation tasks across mobile, desktop, and web platforms.
  • VisualWebBench: A dataset for GUI navigation, specifically used for evaluating generalization to out-of-distribution data.
  • AITW (Android in the Wild): A large-scale dataset for mobile navigation tasks.
  • Mind2Web: A benchmark for web navigation tasks, covering cross-task, cross-website, and cross-domain scenarios.
  • ScreenPR: A benchmark used for evaluating GUI referring capabilities.

The authors also mention using examples from the ScreenSpot dataset for SEED's in-context learning.

Key Insights and Ideas

  • Holistic GUI Understanding is Crucial: The paper emphasizes that moving beyond isolated task performance (like just action grounding) to a comprehensive understanding of GUI structure and element functionality is key for more capable AI agents.
  • Hierarchy Matters: Decomposing GUIs into hierarchical regions (GROIs) and local elements, as done by HSP, provides a more structured and interpretable representation that LVLMs can leverage effectively.
  • Context is King for Descriptions: SEED's approach of using spatial and semantic context to generate element descriptions significantly improves their accuracy and usefulness compared to standalone OCR or visual features.
  • Training-Free is Powerful: TRISHUL's ability to enhance generalist LVLMs without task-specific fine-tuning highlights the potential of leveraging foundational models for complex UI interactions.
  • Cross-Platform Generalization is Achievable: The framework demonstrates strong performance across diverse platforms (mobile, desktop, web), addressing a major limitation of previous methods.

Key Results

TRISHUL demonstrates significant improvements across various benchmarks:

  • Action Grounding: TRISHUL, when paired with LVLMs like GPT-4V and GPT-40, outperforms prior state-of-the-art methods on ScreenSpot and VisualWebBench. For instance, it achieves an overall accuracy of 61.9% with GPT-4V and 72.2% with GPT-40 on ScreenSpot, surpassing specialized models. It also shows robust generalization on VisualWebBench, achieving 68.0% accuracy.
  • GUI Referring: On the ScreenPR benchmark, TRISHUL surpasses the ToL agent in human evaluations, with descriptions generated by TRISHUL being preferred 73% of the time over GPT-40 and 62.8% over the ToL agent.
  • Ablation Studies: Removing SEED or the GROI-based grounding in TRISHUL leads to notable accuracy drops, highlighting the critical role of these components.
  • Modularity: Augmenting existing pipelines (like OmniParser) with TRISHUL's SEED module yields superior results.
  • GPT-40 vs. GPT-4V: GPT-40 generally shows improved performance when paired with SEED, suggesting that better reasoning capabilities in LVLMs enhance the accuracy of generated descriptions.

Limitations and Future Work

  • Mobile GUI Challenges: The paper notes that GROIs are less useful for mobile GUIs due to their less coherent semantic structure and fewer detected GROIs. The Local Element Exhaustiveness (LEE) score is lower for mobile datasets, indicating a bottleneck in the HSP module's ability to detect all relevant local elements.
  • HTML-based Methods: While TRISHUL excels with image-only inputs, it acknowledges that HTML-based methods can still achieve higher element accuracy and step success rates in certain scenarios, as predicting bounding boxes from pixels is more complex than parsing structured HTML.
  • Future Work: The authors suggest leveraging TRISHUL as a verification agent in GUI systems to enable multi-click grounding with enhanced accuracy by sampling multiple action paths. They also point out that improved LVLM reasoning capabilities can further boost performance.

Summary updated

BibTeX
@inproceedings{singh2025trishul,
  author = {Kunal Singh and Shreyas Singh and Mukund Khanna},
  title = {{TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents}},
  booktitle = {Workshops @ CVPR},
  year = {2025},
}
Figure