Modern digital tasks rarely happen in a vacuum. A simple request like "schedule a meeting based on this email" requires an agent to parse text in a mail client, check availability in a calendar, and perhaps look up a contact in a third-party directory. While Large Language Model (LLM) agents have become proficient at navigating individual interfaces, their performance collapses when a task requires crossing the boundaries between different applications.
This "execution gap," as described by Kong et al., represents a fundamental shift in complexity. When an agent moves from a single-app environment to a multi-app workflow, it must manage heterogeneous data formats, maintain state across context switches, and coordinate long-horizon plans.
The Friction of Context Switching
The primary reason agents fail in multi-app scenarios is the loss of information during transitions. In a single application, the UI state is often persistent and visible. However, as Xue et al. observe in their study of Android environments, success rates drop significantly because agents struggle to maintain task-relevant context when navigating diverse interface structures.
This failure often manifests during "message passing." Xu et al. highlight that friction occurs when information retrieved from one platform (like Ubuntu) must be reformatted or correctly applied in another (like Android). If the agent cannot robustly store and retrieve this intermediate data, the workflow breaks. To combat this, Chen et al. argue for more sophisticated long-term memory architectures that prevent the loss of critical information during application switches.
Hierarchical Planning and Tool Orchestration
To solve these complex sequences, researchers are moving toward hierarchical architectures that separate high-level strategy from low-level execution. Instead of a single model trying to handle every click and every logical step, the system is split into a "conductor" and an "executor."
For example, Qin proposes the UFO framework, which uses a dual-agent system to navigate Windows. A high-level agent manages the global plan across different software contexts, while a UI-focused agent executes specific sub-tasks within an application. Similarly, Ahn et al. demonstrate that separating workflow planning from tool execution allows agents to better respect real-world constraints and dependencies between disparate services.
This orchestration is particularly difficult when the tools belong to different domains. Wang et al. note that transitioning between different tool ecosystems—such as moving data from a web browser to a script editor and then to a terminal—introduces domain-specific logic that even advanced models struggle to synchronize.
Benchmarking the Multi-App Reality
The shift toward multi-app orchestration has necessitated a new generation of benchmarks that move beyond "monkey testing" or isolated API calls.
- Real-World Environments: Kong et al. (OSWorld) and Kapoor et al. (OmniACT) provide executable environments across Windows, macOS, and Ubuntu to test generalist assistants.
- Functional Integration: Deng et al. (Mobile-Bench) integrate functional APIs to measure how agents bridge data gaps between multiple apps.
- Cross-Platform Interaction: Xu et al. (CRAB) evaluate agents interacting with multiple operating systems simultaneously, reflecting the multi-device reality of modern work.
These benchmarks consistently show that while agents can handle isolated, low-complexity tasks, they lack the "meta-reasoning" required for integrated workflows. As He et al. point out, the high-dimensional planning required for inter-domain task management remains an open challenge, requiring agents that can not only use tools but also understand the connective tissue between them.
Key Foundational Work
FORWARD: Design Specification Techniques for Do-It-Yourself Application Platforms (2009) — contributed 1 concept elements to this direction.
A schema based approach to dialog control (1998) — contributed 1 concept elements to this direction.