Abstract & Introduction
Autonomous Graphical User Interface (GUI) agents often struggle with multi-step tasks due to constrained context windows and static policies that fail to adapt to dynamic environments. To address these limitations, researchers have introduced the Self-Evolving GUI Agent (SE-GA). By combining a hierarchical memory system with a self-improving training pipeline, SE-GA mimics human cognitive structures and learns from its own interactions. This approach allows the agent to achieve state-of-the-art performance on major benchmarks, significantly outperforming static models in dynamic, real-world environments.
The Challenge: Why GUI Agents Fail
Building an AI agent that can navigate a smartphone or website as seamlessly as a human presents two primary hurdles:
- The "Goldfish" Problem: Most agents have limited context windows. In a 30-step task, they often forget actions taken in early steps, leading to repetitive errors or "attention dilution" where they lose track of the ultimate goal.
- Static Policies: Standard agents are trained on fixed datasets and then frozen. They cannot adapt to new app layouts or learn from recent mistakes without undergoing complete retraining.
Because real-world GUIs are partially observable and highly dynamic, an agent that cannot remember its history or evolve based on experience remains a brittle tool rather than a reliable assistant.
Methodology: The SE-GA Framework
SE-GA overcomes these limitations through two core components:
1. Test-Time Memory Extension (TTME)
Instead of relying solely on the current screenshot and a limited context window, SE-GA uses a hierarchical memory repository inspired by human cognition:
- Episodic Memory: A sliding window of the most recent actions that keeps the agent focused on the immediate next step.
- Semantic Memory: A "rulebook" of general digital logic (e.g., understanding that a user must log in before checking out).
- Experiential Memory: A library of successful past trajectories. When faced with a new task, the agent retrieves similar successful experiences to guide its reasoning.
2. Memory-Augmented Self-Evolution (MASE)
MASE is the training pipeline that allows the agent to continuously improve. It operates in two stages:
- Stage I (Grounding): Supervised fine-tuning helps the model understand the relationship between text instructions and UI elements.
- Stage II (Self-Evolution): The agent interacts with environments and uses Group Relative Policy Optimization (GRPO) to refine its policy.
A standout feature of MASE is the Hindsight Goal-Shifting Mechanism. If the agent fails a complex task (e.g., "Book a flight and save the receipt") but successfully completes a sub-task (e.g., "Open the airline app"), the system relabels that failure as a success for the sub-task. This transforms every interaction—even failures—into a valuable learning signal.
Key Insights
- Structured Memory Outperforms Larger Context: Simply increasing the context window is insufficient; agents need structured memory. By separating immediate history from general app logic, the agent avoids being overwhelmed by irrelevant data.
- Failures as Hidden Successes: The Hindsight Goal-Shifting mechanism elegantly solves the "sparse reward" problem. By rewarding successful sub-steps, the agent learns much faster than it would if it only received a reward at the very end of a long task.
- Adaptive Clipping: During reinforcement learning, a dynamic clipping strategy allows the model to make large learning leaps early on, while forcing it to be more precise and stable as it nears peak performance.
Experimental Results
SE-GA was evaluated across diverse benchmarks, consistently outperforming both generalist models (like GPT-4o) and specialized GUI agents:
- ScreenSpot (Static Grounding): Achieved an average success rate of 89.0%, surpassing the much larger UI-TARS-72B model.
- AndroidControl-High (Planning & Execution): Reached a 75.8% success rate, setting a new state-of-the-art for 7B-parameter models.
- AndroidWorld (Dynamic Environments): In this live environment testing real-world latency and layout changes, SE-GA achieved a 39.0% success rate, significantly higher than the baseline Qwen2.5-VL (25.5%).
- Ablation on Memory: Removing the TTME module caused a 12.4% drop in success on long-horizon tasks, proving that structured memory is the engine behind complex planning.
Limitations and Future Work
The primary limitation of SE-GA is Memory Retrieval Efficiency. As the agent's experiential library grows, searching through thousands of past trajectories could introduce latency, potentially making the agent sluggish in real-time applications.
Future research directions include:
- Scaling Data: Expanding beyond the current 4K trajectories to a much larger corpus.
- Hierarchical Decomposition: Teaching the agent to explicitly break down "ultra-long" workflows into smaller, manageable sub-goals.
- Cross-Platform Transfer: Investigating whether an agent evolved on Android can seamlessly transfer its skills to a Windows desktop or web browser.
Summary updated