Introduction
Graphical User Interface (GUI) agents represent a significant frontier in human-device interaction, promising to automate complex workflows across diverse software environments. However, current agents often struggle when encountering unfamiliar applications. This lack of generalization stems from two fundamental limitations in existing research: developer-induced structural variations across apps and a narrow focus on simple navigation tasks.
To address these challenges, we introduce GUI-Xplore, a dataset and framework designed to move beyond generalized pattern matching toward an "explore-then-reason" paradigm. By providing agents with a single exploration video of an unfamiliar application, we enable them to build a structural understanding of the environment before attempting to execute tasks. Our proposed Xplore-Agent leverages this approach to achieve a 10% performance improvement in new environments compared to state-of-the-art methods.
The Generalization Gap in GUI Agents
While Large Vision-Language Models (LVLMs) have enhanced the capabilities of GUI agents, they frequently fail in zero-shot scenarios for two primary reasons:
- Structural Variations: Software design is highly subjective. The logic required to navigate a payment screen in one app (e.g., PayPal) rarely translates directly to another (e.g., WeChat). Existing datasets lack the app-specific context necessary for agents to navigate these "developer-driven" differences.
- Narrow Task Focus: Most benchmarks prioritize simple automation, such as "click the search bar." These tasks do not test whether an agent understands the broader architecture of an application or can handle complex user intentions that require multi-step reasoning.
GUI-Xplore aims to provide agents with a "map" of the application through a preliminary scouting phase, bridging the gap between general knowledge and specific application logic.
The GUI-Xplore Dataset
The GUI-Xplore dataset is significantly more comprehensive than previous benchmarks, focusing on depth and structural complexity. It features:
- Scale and Diversity: 312 applications across 33 categories, totaling 115 hours of exploratory video.
- Long-Form Context: Unlike datasets like AITW, which use short clips averaging 6.5 frames, GUI-Xplore’s videos average 21.73 minutes. This allows for a complete representation of an app's functional architecture.
- Hierarchical Tasks: The dataset includes 32,569 Q&A pairs categorized into five levels of reasoning:
- Application Overview: Summarizing core functionality.
- Page Analysis: Describing specific screen functions.
- Application Usage: Pathfinding to specific goals.
- Action Recall: Identifying specific historical interactions.
- Action Sequence Verification: Validating the correct order of operations.
Xplore-Agent: A Two-Stage Framework
To utilize the GUI-Xplore dataset effectively, we developed Xplore-Agent, a framework that transforms raw exploration video into actionable intelligence through two stages.
Stage 1: Action-aware GUI Modeling
Processing high-resolution video is computationally expensive. Xplore-Agent uses Luminance Difference (Y-Diff) to extract keyframes, identifying the exact moments user actions begin and end. These visual frames are then converted into simplified textual View Hierarchies (VH) and action descriptions. This abstraction reduces the data footprint from heavy video files to lightweight, searchable text summaries.
Stage 2: Graph-Guided Environment Reasoning
Instead of processing the app history as a linear stream, the agent clusters similar screens to construct a GUI Transition Graph. In this graph, nodes represent functional pages (e.g., "Settings" or "Cart") and edges represent the actions that transition the user between them. When a task is assigned, the LLM "walks" this graph to find the optimal sequence of actions, transforming a temporal problem into a more manageable topological one.
Key Results and Insights
Our experiments demonstrate that the "one-shot" exploration video acts as a powerful inductive prior. Key findings include:
- Superior Generalization: In unfamiliar environments, Xplore-Agent achieved a 10% improvement in Element Accuracy and Step Success Rate over SOTA models like CogAgent and AUTO-UI.
- Reasoning Accuracy: On the multi-task benchmark, Xplore-Agent reached an average accuracy of 64.24%, outperforming GPT-4V (59.39%) and specialized video-reasoning models like VideoTree (57.97%).
- Efficiency through Abstraction: Ablation studies revealed that using a transition graph reduced token usage from over 5 million (for raw video) to approximately 45,000, while simultaneously increasing accuracy. This suggests that structured abstraction is more effective for GUI reasoning than raw data density.
Limitations and Future Work
While GUI-Xplore marks a significant step forward, several challenges remain. Currently, the framework is evaluated on its ability to generate textual plans and answers; future work will focus on integrating it into a "closed-loop" system capable of real-time execution on live devices. Additionally, the use of long-form exploration videos necessitates further research into privacy-preserving data collection and reducing the latency associated with graph construction.
By shifting the focus from simple navigation to structured exploration and reasoning, GUI-Xplore provides a robust foundation for the next generation of generalizable GUI agents.
Summary updated