GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration

Yuchen Sun, Shanhui Zhao, Tao Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, Chongyang Zhang

Introduction

Graphical User Interface (GUI) agents represent a significant frontier in human-device interaction, promising to automate complex workflows across diverse software environments. However, current agents often struggle when encountering unfamiliar applications. This lack of generalization stems from two fundamental limitations in existing research: developer-induced structural variations across apps and a narrow focus on simple navigation tasks.

To address these challenges, we introduce GUI-Xplore, a dataset and framework designed to move beyond generalized pattern matching toward an "explore-then-reason" paradigm. By providing agents with a single exploration video of an unfamiliar application, we enable them to build a structural understanding of the environment before attempting to execute tasks. Our proposed Xplore-Agent leverages this approach to achieve a 10% performance improvement in new environments compared to state-of-the-art methods.

The Generalization Gap in GUI Agents

While Large Vision-Language Models (LVLMs) have enhanced the capabilities of GUI agents, they frequently fail in zero-shot scenarios for two primary reasons:

  1. Structural Variations: Software design is highly subjective. The logic required to navigate a payment screen in one app (e.g., PayPal) rarely translates directly to another (e.g., WeChat). Existing datasets lack the app-specific context necessary for agents to navigate these "developer-driven" differences.
  2. Narrow Task Focus: Most benchmarks prioritize simple automation, such as "click the search bar." These tasks do not test whether an agent understands the broader architecture of an application or can handle complex user intentions that require multi-step reasoning.

GUI-Xplore aims to provide agents with a "map" of the application through a preliminary scouting phase, bridging the gap between general knowledge and specific application logic.

The GUI-Xplore Dataset

The GUI-Xplore dataset is significantly more comprehensive than previous benchmarks, focusing on depth and structural complexity. It features:

  • Scale and Diversity: 312 applications across 33 categories, totaling 115 hours of exploratory video.
  • Long-Form Context: Unlike datasets like AITW, which use short clips averaging 6.5 frames, GUI-Xplore’s videos average 21.73 minutes. This allows for a complete representation of an app's functional architecture.
  • Hierarchical Tasks: The dataset includes 32,569 Q&A pairs categorized into five levels of reasoning:
  • Application Overview: Summarizing core functionality.
  • Page Analysis: Describing specific screen functions.
  • Application Usage: Pathfinding to specific goals.
  • Action Recall: Identifying specific historical interactions.
  • Action Sequence Verification: Validating the correct order of operations.

Xplore-Agent: A Two-Stage Framework

To utilize the GUI-Xplore dataset effectively, we developed Xplore-Agent, a framework that transforms raw exploration video into actionable intelligence through two stages.

Stage 1: Action-aware GUI Modeling

Processing high-resolution video is computationally expensive. Xplore-Agent uses Luminance Difference (Y-Diff) to extract keyframes, identifying the exact moments user actions begin and end. These visual frames are then converted into simplified textual View Hierarchies (VH) and action descriptions. This abstraction reduces the data footprint from heavy video files to lightweight, searchable text summaries.

Stage 2: Graph-Guided Environment Reasoning

Instead of processing the app history as a linear stream, the agent clusters similar screens to construct a GUI Transition Graph. In this graph, nodes represent functional pages (e.g., "Settings" or "Cart") and edges represent the actions that transition the user between them. When a task is assigned, the LLM "walks" this graph to find the optimal sequence of actions, transforming a temporal problem into a more manageable topological one.

Key Results and Insights

Our experiments demonstrate that the "one-shot" exploration video acts as a powerful inductive prior. Key findings include:

  • Superior Generalization: In unfamiliar environments, Xplore-Agent achieved a 10% improvement in Element Accuracy and Step Success Rate over SOTA models like CogAgent and AUTO-UI.
  • Reasoning Accuracy: On the multi-task benchmark, Xplore-Agent reached an average accuracy of 64.24%, outperforming GPT-4V (59.39%) and specialized video-reasoning models like VideoTree (57.97%).
  • Efficiency through Abstraction: Ablation studies revealed that using a transition graph reduced token usage from over 5 million (for raw video) to approximately 45,000, while simultaneously increasing accuracy. This suggests that structured abstraction is more effective for GUI reasoning than raw data density.

Limitations and Future Work

While GUI-Xplore marks a significant step forward, several challenges remain. Currently, the framework is evaluated on its ability to generate textual plans and answers; future work will focus on integrating it into a "closed-loop" system capable of real-time execution on live devices. Additionally, the use of long-form exploration videos necessitates further research into privacy-preserving data collection and reducing the latency associated with graph construction.

By shifting the focus from simple navigation to structured exploration and reasoning, GUI-Xplore provides a robust foundation for the next generation of generalizable GUI agents.

Summary updated

Abstract

GUI agents hold significant potential to enhance the experience and efficiency of human-device interaction. However, current methods face challenges in generalizing across applications (apps) and tasks, primarily due to two fundamental limitations in existing datasets. First, these datasets overlook developer-induced structural variations among apps, limiting the transferability of knowledge across diverse software environments. Second, many of them focus solely on navigation tasks, which restricts their capacity to represent comprehensive software architectures and complex user interactions. To address these challenges, we introduce GUI-Xplore, a dataset meticulously designed to enhance cross-application and cross-task generalization via an exploration-and-reasoning framework. GUI-Xplore integrates pre-recorded exploration videos providing contextual insights, alongside five hierarchically structured downstream tasks designed to comprehensively evaluate GUI agent capabilities. To fully exploit GUI-Xplore's unique features, we propose Xplore-Agent, a GUI agent framework that combines Action-aware GUI Modeling with Graph-Guided Environment Reasoning. Further experiments indicate that Xplore-Agent achieves a 10% improvement over existing methods in unfamiliar environments, yet there remains significant potential for further enhancement towards truly generalizable GUI agents.
BibTeX
@inproceedings{sun2025gui,
  author = {Yuchen Sun and Shanhui Zhao and Tao Yu and Hao Wen and Samith Va and Mengwei Xu and Yuanchun Li and Chongyang Zhang},
  title = {{GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=rcdf3n4MxA},
}
Figure