Abstract
Large Language Model (LLM)-based agents for Graphical User Interfaces (GUIs) often fail during long-horizon tasks due to context window limitations and the loss of critical information over time. We introduce AgentProg, a framework that treats agent interaction history as a structured program. By utilizing a "fuzzy" domain-specific language and a global belief state mechanism, AgentProg manages context efficiently, enabling agents to complete complex, multi-step tasks across mobile applications.
Introduction
Building intelligent agents capable of navigating mobile applications is a cornerstone of autonomous AI. However, current agents hit a "context wall" when tasks extend over dozens or hundreds of steps. Standard approaches suffer from context explosion, where history consumes excessive tokens; information loss, where critical details are discarded by compression; and partial observability, where agents lose track of the environment's internal state.
Methodology: The Program-Guided Approach
AgentProg reframes task execution as a running computer program, employing three core components to maintain focus and efficiency.
Semantic Task Program (STP)
The STP is a domain-specific language (DSL) that bridges the gap between rigid code and flexible natural language. It uses programming constructs like loops (iterate over {task_list}), conditionals (if {age} < 18), and variables (record as {phone_number}) while keeping the underlying instructions in "soft" natural language. This allows the agent to follow a logical plan while adapting to UI variations.
Program-Guided Context Management
Instead of maintaining a flat action history, AgentProg organizes interactions into an Execution Tree. It employs Path-Based Pruning to remove irrelevant branches (e.g., completed loop iterations) and uses Variables as persistent anchors for critical data, ensuring information remains accessible regardless of task length.
Global Belief State
Inspired by the Belief MDP framework, this module maintains a "mental model" of the environment through a Predict-Verify-Align cycle. The agent predicts the expected UI state, verifies it against actual screenshots, and triggers recovery routines if a "Belief-Reality Gap" (such as an app crash) is detected.
Experimental Evaluation
The researchers evaluated AgentProg on AndroidWorld (116 tasks) and a new, more challenging suite called AW-Extend. AW-Extend focuses on long-horizon capabilities, including compositional multi-app workflows and iterative tasks requiring repeated sub-task execution.
Key Results
AgentProg demonstrates significant improvements in both success rate and efficiency:
- State-of-the-Art Performance: Achieved a 78.0% success rate on AndroidWorld, a 4.7% improvement over previous benchmarks.
- Long-Horizon Robustness: On the AW-Extend benchmark, AgentProg reached 68.4%, nearly doubling the performance of the next best baseline (36.8%).
- Context Efficiency: Unlike standard agents whose memory usage grows linearly, AgentProg maintains a nearly constant dynamic context length (~9k tokens) even during 50+ step tasks.
- Ablation Insights: Removing the Global Belief State caused the success rate to drop from 78% to 53.9%, proving the necessity of active environment monitoring.
Conclusion and Future Work
AgentProg demonstrates that treating interaction history as a structured program is a powerful paradigm for long-horizon GUI agents. While the current implementation faces challenges regarding latency and prompt overhead, future work involving fine-tuned specialized models for STP generation could further optimize performance and responsiveness.
Summary updated