Abstract
Recently, mobile AI agents have gained increasing attention for their ability to interact with mobile devices across multiple steps to solve complex tasks. However, existing agents often focus on the most task-relevant elements at each individual step, leading to local optimal solutions that ignore the overall GUI flow. To address this, we introduce ReachAgent, a two-stage framework that decomposes tasks into "page reaching" and "page operation" subtasks. By utilizing a new training dataset, MobileReach, and a reward-based preference system, ReachAgent significantly improves task-completion abilities. Experimental results demonstrate that ReachAgent outperforms state-of-the-art (SOTA) agents, improving step-level IoU and Text Accuracy by 7.12% and 7.69%, respectively.
Introduction
Current mobile AI agents—models designed to navigate applications like a human user—frequently suffer from a "greedy" search problem. When tasked with a multi-step instruction (e.g., "Add a white Xiaomi 14 to my cart"), these agents tend to search for the most task-relevant keywords on the current screen. If the agent is on a homepage and does not immediately see the target item, it may click irrelevant banners or become stuck in a loop because it fails to recognize the necessity of a "Reach" subtask, such as using a search bar or navigating categories.
Existing agents prioritize single-step accuracy but often overlook the global GUI flow required to complete a multi-step job. ReachAgent addresses this by explicitly training the model to distinguish between navigating to a destination and performing operations once the destination is reached.
The ReachAgent Framework
ReachAgent is built on a MobileVLM backbone and utilizes a two-stage training process to refine its navigation and interaction logic.
Stage 1: Supervised Fine-Tuning (SFT)
The model is first trained on the MobileReach dataset to acquire three distinct skills:
- General Navigation: Understanding basic app structures.
- Page Reaching: Learning the sequences required to arrive at specific functional pages.
- Page Operation: Executing specific tasks (e.g., selecting attributes, clicking "buy") once the correct page is reached.
Stage 2: Reinforcement Learning via DPO
To improve efficiency, the framework employs Direct Policy Optimization (DPO) using a 4-level reward function to rank GUI flows:
- Golden: The most efficient, shortest path to success.
- Longer: A successful path that includes unnecessary steps.
- Incomplete: A path that failed to reach the goal.
- Invalid: A path containing impossible actions, such as clicking non-existent UI elements.
Action Alignment Mechanism
A key technical innovation in ReachAgent is the Action Alignment Mechanism. Rather than predicting arbitrary X/Y coordinates on a raw screenshot, ReachAgent targets the center points of UI elements identified in the application's XML code. This reduces the search space and ensures the agent interacts with valid, functional components of the interface.
Datasets: MobileReach
The researchers developed MobileReach, a comprehensive dataset derived from "Mobile3M" covering 49 popular apps. The dataset includes:
- Page Navigation: ~54k flows for moving between screens.
- Page Reaching: ~68k flows focused on reaching specific destinations.
- Page Operation: ~76k flows for executing tasks on target pages.
Additionally, the model was evaluated on the Auto-UI benchmark to test its generalization capabilities on unseen tasks.
Results and Insights
ReachAgent demonstrates a significant performance leap over previous SOTA mobile agents:
- Step-Level Accuracy: Improved Intersection over Union (IoU) by 7.12% and Text Accuracy by 7.69%.
- Task-Level Success: Achieved a 4.72% and 4.63% boost in completing entire multi-step flows correctly.
- Generalization: Even on the Auto-UI dataset, where it was not specifically trained on subtasks, ReachAgent outperformed all baselines, proving that the "Reach and Operate" logic is a universally effective strategy for mobile interaction.
Conclusion
ReachAgent highlights that subtask decomposition is critical for mobile AI. By teaching the model that reaching a page is a valid intermediate goal, the framework prevents "greedy" errors. Furthermore, by favoring "Golden" paths through DPO, ReachAgent learns to be an efficient "power user" rather than just a functional one. Future work will focus on automated exploration to build deeper datasets and further refining reward models to capture nuanced user preferences.
Summary updated