Abstract
GUI agents hold significant potential to enhance human-device interaction, yet they struggle to generalize across unfamiliar applications and complex tasks. We introduce GUI-Xplore, a dataset featuring 312 apps and 115 hours of exploration videos, designed to facilitate cross-application generalization. Alongside this, we propose Xplore-Agent, a framework that utilizes a "GUI Transition Graph" to reason about app structures. Our experiments show that Xplore-Agent improves performance by 10% in zero-shot environments compared to existing state-of-the-art models.
Introduction: The Generalization Gap
While Large Language Models (LLMs) have revolutionized digital automation, GUI agents—virtual assistants capable of navigating software interfaces—face two primary hurdles: structural variation and task narrowness.
Every application is architected differently; for instance, a "settings" menu in a financial app may be located and structured entirely differently than one in a social media app. Most current agents are trained on specific datasets and fail when introduced to new environments because they lack app-specific context. Furthermore, existing benchmarks often focus on simple navigation tasks, neglecting the complex reasoning required to understand an app's underlying logic.
The Xplore-Agent Framework
To address these challenges, we propose Xplore-Agent, a two-stage framework that treats a single exploration video as a "knowledge prior." Humans don't just guess when using a new app; we explore. Xplore-Agent mimics this strategy through two components:
- Action-aware GUI Modeling: The system extracts keyframes from exploration videos using luminance difference (Y-Diff) to identify screen changes. A Vision-Language Model (LVLM) then converts these frames into simplified View Hierarchies (VH) and textual action descriptions.
- Graph-Guided Environment Reasoning: To manage the non-linear nature of modern apps, the agent clusters similar screens to build a GUI Transition Graph. This graph serves as a map, allowing an LLM to reason about the app's structure and logic efficiently without processing a linear video timeline.
The GUI-Xplore Dataset
The core contribution of this work is the GUI-Xplore Dataset, which provides a more comprehensive benchmark for GUI agents than previous efforts like Mind2Web or AITW. It stands out due to its scale (312 apps across 33 categories) and the depth of its exploration videos, which average over 21 minutes in length.
Beyond simple navigation, the benchmark evaluates agents across five distinct hierarchical tasks:
- Application Overview: Summarizing the app's purpose.
- Page Analysis: Describing specific screen functions.
- Application Usage: Inferring the sequence needed to reach a goal.
- Action Recall: Identifying when a specific action occurred in the video.
- Action Sequence Verification: Checking if a proposed path is logically valid.
Experimental Results
The Xplore-Agent baseline significantly outperformed existing state-of-the-art (SOTA) models in cross-app generalization:
- Accuracy: In unfamiliar environments, Xplore-Agent achieved 30.73% Element Accuracy, nearly doubling the performance of CogAgent (17.18%).
- Success Rate: It reached a 30.39% Step Success Rate (StepSR), compared to 15.80% for the next best model.
- Efficiency: The use of the GUI Transition Graph reduced token usage by over 90% compared to processing raw video frames, while simultaneously increasing accuracy.
Conclusion and Future Work
GUI-Xplore demonstrates that "watching" an exploration video provides a critical knowledge prior for agents entering unfamiliar environments. However, limitations remain. Current agents rely on textual descriptions rather than raw system events, and dynamic content (like live social feeds) remains a challenge for static graph-based modeling. Future research should focus on interactive agents that can explore and learn autonomously in real-time.
Summary updated