Abstract
Recently, mobile AI agents have gained increasing attention. Given a task, mobile AI agents can interact with mobile devices in multiple steps and finally form a GUI flow that solves the task. However, existing agents tend to focus on most task-relevant elements at each step, leading to local optimal solutions and ignoring the overall GUI flow. To address this issue, we constructed a training dataset called MobileReach, which breaks the task into page reaching and operation subtasks. Furthermore, we propose ReachAgent, a two-stage framework that focuses on improving its task-completion abilities. It utilizes the page reaching and page operation subtasks, along with reward-based preference GUI flows, to further enhance the agent. Experimental results show that ReachAgent significantly improves the Intersection over Union (IoU) Accuracy and Text Accuracy by 7.12% and 7.69% on the step-level and 4.72% and 4.63% on the task-level compared to the SOTA agent.
Introduction
Current mobile AI agents, typically powered by Vision-Language Models (VLMs), often suffer from "tunnel vision." When tasked with a multi-step objective, such as "Add a white Xiaomi 14 to the shopping cart," these agents frequently prioritize elements visible on the immediate screen. If the target item is not explicitly displayed on the homepage, the agent may default to random interactions or enter infinite loops. This phenomenon, known as falling into local optima, stems from a lack of understanding of the broader "GUI flow"—the sequence of intermediate pages (e.g., search results, category menus) required to reach a final goal.
To overcome these limitations, we propose decomposing complex tasks into structured subtasks: Page Reaching and Page Operation. This approach encourages the agent to navigate through the application hierarchy systematically rather than searching for immediate gratification.
Methodology
ReachAgent is a two-stage framework designed to enhance task-completion capabilities through structured learning and preference optimization.
1. Action Alignment Mechanism
To simplify the action space, we introduce an Action Alignment mechanism. Instead of predicting arbitrary pixel coordinates, the agent is constrained to interact with the center points of UI elements or scroll along their central axes. This aligns the agent's actions with the underlying XML structure of the application, allowing the VLM to focus on high-level reasoning rather than precise coordinate estimation.
2. Two-Stage Training Framework
The framework consists of two primary training phases:
- Stage 1: Supervised Fine-Tuning (SFT): The model is trained on three distinct task types: standard multi-step Page Navigation, Page Reaching (navigating from the home page to a target), and Page Operation (performing actions on a specific target page). This multi-task training provides the model with a "mental map" of app hierarchies.
- Stage 2: Reinforcement Learning (DPO): We utilize Direct Policy Optimization (DPO) to refine agent behavior using a preference-based reward system. GUI flows are categorized into four levels: Golden (optimal path), Longer (successful but redundant), Incomplete (failed to reach goal), and Invalid (unexecutable actions). This ranking teaches the agent to prioritize efficiency and avoid "wandering" behavior.
Experimental Evaluation
We introduced MobileReach, a comprehensive dataset derived from the Mobile3M graph structure, containing over 190,000 GUI flows across navigation, reaching, and operation tasks, along with 259,742 preference pairs for RL.
Key Results
ReachAgent demonstrates significant performance gains over state-of-the-art baselines:
- MobileReach Benchmarks: On the MobileReach dataset, ReachAgent improved step-level IoU by 7.12% and Text Accuracy by 7.69% compared to MobileVLM. Task-level success rates saw a 6.59% improvement.
- Generalization: When evaluated on the public Auto-UI dataset, ReachAgent achieved an overall accuracy of 75.27%, outperforming general-purpose models like GPT-4o and specialized SFT models.
Qualitative analysis confirms that ReachAgent effectively identifies when to switch from browsing to searching, avoiding the common failure mode of getting stuck in repetitive scrolling loops.
Conclusion
ReachAgent addresses the "local optima" problem in mobile AI agents by decomposing tasks into reaching and operating subtasks. By combining this decomposition with action alignment and preference-based reinforcement learning, the agent achieves superior efficiency and success rates. While currently trained on 49 common apps, future work will explore cross-app task execution and handling more complex user-state dependencies.
Summary updated