SE-GA: Memory-Augmented Self-Evolution for GUI Agents

Shilong Jin, Lanjun Wang, Zhuosheng Zhang

Abstract & Introduction

Autonomous Graphical User Interface (GUI) agents often struggle with multi-step tasks due to constrained context windows and static policies that fail to adapt to dynamic environments. To address these limitations, researchers have introduced the Self-Evolving GUI Agent (SE-GA). By combining a hierarchical memory system with a self-improving training pipeline, SE-GA mimics human cognitive structures and learns from its own interactions. This approach allows the agent to achieve state-of-the-art performance on major benchmarks, significantly outperforming static models in dynamic, real-world environments.

The Challenge: Why GUI Agents Fail

Building an AI agent that can navigate a smartphone or website as seamlessly as a human presents two primary hurdles:

  1. The "Goldfish" Problem: Most agents have limited context windows. In a 30-step task, they often forget actions taken in early steps, leading to repetitive errors or "attention dilution" where they lose track of the ultimate goal.
  2. Static Policies: Standard agents are trained on fixed datasets and then frozen. They cannot adapt to new app layouts or learn from recent mistakes without undergoing complete retraining.

Because real-world GUIs are partially observable and highly dynamic, an agent that cannot remember its history or evolve based on experience remains a brittle tool rather than a reliable assistant.

Methodology: The SE-GA Framework

SE-GA overcomes these limitations through two core components:

1. Test-Time Memory Extension (TTME)

Instead of relying solely on the current screenshot and a limited context window, SE-GA uses a hierarchical memory repository inspired by human cognition:

  • Episodic Memory: A sliding window of the most recent actions that keeps the agent focused on the immediate next step.
  • Semantic Memory: A "rulebook" of general digital logic (e.g., understanding that a user must log in before checking out).
  • Experiential Memory: A library of successful past trajectories. When faced with a new task, the agent retrieves similar successful experiences to guide its reasoning.

2. Memory-Augmented Self-Evolution (MASE)

MASE is the training pipeline that allows the agent to continuously improve. It operates in two stages:

  • Stage I (Grounding): Supervised fine-tuning helps the model understand the relationship between text instructions and UI elements.
  • Stage II (Self-Evolution): The agent interacts with environments and uses Group Relative Policy Optimization (GRPO) to refine its policy.

A standout feature of MASE is the Hindsight Goal-Shifting Mechanism. If the agent fails a complex task (e.g., "Book a flight and save the receipt") but successfully completes a sub-task (e.g., "Open the airline app"), the system relabels that failure as a success for the sub-task. This transforms every interaction—even failures—into a valuable learning signal.

Key Insights

  • Structured Memory Outperforms Larger Context: Simply increasing the context window is insufficient; agents need structured memory. By separating immediate history from general app logic, the agent avoids being overwhelmed by irrelevant data.
  • Failures as Hidden Successes: The Hindsight Goal-Shifting mechanism elegantly solves the "sparse reward" problem. By rewarding successful sub-steps, the agent learns much faster than it would if it only received a reward at the very end of a long task.
  • Adaptive Clipping: During reinforcement learning, a dynamic clipping strategy allows the model to make large learning leaps early on, while forcing it to be more precise and stable as it nears peak performance.

Experimental Results

SE-GA was evaluated across diverse benchmarks, consistently outperforming both generalist models (like GPT-4o) and specialized GUI agents:

  • ScreenSpot (Static Grounding): Achieved an average success rate of 89.0%, surpassing the much larger UI-TARS-72B model.
  • AndroidControl-High (Planning & Execution): Reached a 75.8% success rate, setting a new state-of-the-art for 7B-parameter models.
  • AndroidWorld (Dynamic Environments): In this live environment testing real-world latency and layout changes, SE-GA achieved a 39.0% success rate, significantly higher than the baseline Qwen2.5-VL (25.5%).
  • Ablation on Memory: Removing the TTME module caused a 12.4% drop in success on long-horizon tasks, proving that structured memory is the engine behind complex planning.

Limitations and Future Work

The primary limitation of SE-GA is Memory Retrieval Efficiency. As the agent's experiential library grows, searching through thousands of past trajectories could introduce latency, potentially making the agent sluggish in real-time applications.

Future research directions include:

  1. Scaling Data: Expanding beyond the current 4K trajectories to a much larger corpus.
  2. Hierarchical Decomposition: Teaching the agent to explicitly break down "ultra-long" workflows into smaller, manageable sub-goals.
  3. Cross-Platform Transfer: Investigating whether an agent evolved on Android can seamlessly transfer its skills to a Windows desktop or web browser.

Summary updated

Abstract

Autonomous Graphical User Interface (GUI) agents often struggle with multi-step tasks due to constrained context windows and static policies that fail to adapt to dynamic environments. To address these limitations, this work proposes the Self-Evolving GUI Agent (SE-GA), a novel framework that integrates hierarchical memory structures with an iterative self-improvement mechanism. At the core of our approach is Test-Time Memory Extension (TTME), which facilitates long-term planning by dynamically retrieving episodic, semantic, and experiential memories to provide salient contexts during inference. To ensure continuous learning, we introduce Memory-Augmented Self-Evolution (MASE), which is a training pipeline that adopts the data collected by TTME to stabilize and enhance the agent's foundational policy. Extensive evaluations across both offline and online benchmarks demonstrate SE-GA achieves state-of-the-art performance, reaching success rates of 89.0\% on ScreenSpot and 75.8\% on the challenging AndroidControl-High dataset. Furthermore, significant improvements on the AndroidWorld benchmark highlight the superior generalization to dynamic environments.
BibTeX
@inproceedings{jin2026se,
  author = {Shilong Jin and Lanjun Wang and Zhuosheng Zhang},
  title = {{SE-GA: Memory-Augmented Self-Evolution for GUI Agents}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=9S9V5ntQJM},
}
Figure