Abstract
Automating user interface (UI) tasks on mobile devices requires a balance between generalizability and computational efficiency. UINav is a demonstration-based framework designed to train lightweight UI automation agents capable of running directly on smartphones. By integrating a "referee model" to guide error-driven data collection, utilizing "macro actions" to handle system latency, and employing robust data augmentation, UINav achieves over 90% task success rates. Crucially, it outperforms existing state-of-the-art approaches while requiring significantly fewer human demonstrations and maintaining a minimal memory footprint.
Introduction
Automating complex UI tasks—such as ordering food or managing calendar invites—has historically faced a trade-off. Hard-coded scripts are brittle and break during app updates, while Large Language Models (LLMs) are often too computationally expensive for on-device use and require massive datasets. UINav identifies a "sweet spot," providing a robust agent small enough for near-instantaneous inference on low-end mobile devices without relying on server-side backends.
Methodology
UINav utilizes an encoder-decoder architecture enhanced by several practical layers designed for real-world mobile environments:
- Macro Actions: Instead of predicting low-level coordinate clicks, the agent predicts "macro actions" (e.g.,
focus_and_type). These atomic programs handle OS-level nuances like waiting for cursors or validating element visibility, reducing the complexity of the state space the model must learn. - The Referee Model: A secondary RNN-based model, the "referee," evaluates task status (success, failure, or pending). This allows for an error-driven development loop where the system identifies specific failure points, prompting human teachers for targeted demonstrations rather than redundant data.
- Data Augmentation and Masking: To improve generalization, UINav generates synthetic demonstrations by randomizing non-critical UI elements (colors, positions). It also uses "utterance masking" to replace specific variables in instructions (e.g., names or numbers) with placeholders, preventing overfitting.
Experimental Setup and Datasets
The researchers evaluated UINav using two primary benchmarks:
- MoTIF Dataset: A public benchmark for goal-oriented UI tasks, used to compare UINav against baselines like Seq2Seq and MOCA.
- Internal Dataset: A collection of 43 diverse tasks (e.g., "clear browser cache," "send a WhatsApp message") spanning 128 Android apps and websites, comprising over 3,600 human demonstrations.
Key Results and Insights
UINav demonstrates significant improvements in both efficiency and performance:
- Task Accuracy: On the MoTIF benchmark, UINav achieved 68.3% task accuracy, more than doubling the performance of the Seq2Act baseline (32.4%).
- On-Device Efficiency: The agent and referee models are remarkably small, occupying only 1.3MB and 1.8MB, respectively. Total inference latency remains low, with the core agent taking only 2ms–5ms, ensuring a responsive user experience even on low-end hardware.
- Scaling with Demonstrations: The system reaches 70% accuracy with as few as 10 demonstrations per task. With a larger pool of data, success rates on complex Android tasks exceed 90%.
- Multi-task Transfer: Results indicate that a single multi-task agent outperforms multiple specialist agents. This suggests the model learns a general "grammar" of mobile UI interactions that transfers across different applications.
Discussion and Limitations
While UINav represents a leap in on-device UI automation, some limitations remain. The current architecture is "memory-less," making decisions based solely on the current screen state. Tasks requiring information from previous screens may require future iterations to include recurrent states or input padding. Additionally, the system relies on accessibility trees; apps that do not expose this metadata (such as certain games or custom WebViews) remain a challenge. Future work will explore expanding the action space to include deep-links and direct API calls to further enhance capability and speed.
Summary updated