While Large Language Models (LLMs) have become proficient at using individual tools—like searching the web or writing a single code snippet—they face a "performance cliff" when tasks require moving across software boundaries. Real-world productivity rarely happens in a single window; it involves extracting data from a spreadsheet, summarizing it in a document, and then emailing a specific contact. This transition between disparate digital environments is known as cross-application orchestration.
The Coordination Gap
The fundamental challenge is that most agents are trained and evaluated in "silos." In a single-application environment, the state is stable and the action space is consistent. However, as X. R. Chen and colleagues observe in OS-MAP, success rates plummet when moving from single-app tasks to multi-app orchestration.
This failure is rarely about the model's ability to click a button. Instead, it is a failure of context integrity. When an agent moves from a mobile app to a desktop environment, or even between two different websites, it must "carry" information across heterogeneous interfaces. Tianqi Xu and the team behind CRAB highlight that agents frequently fail during this "message passing" phase, where data retrieved from one platform is lost or incorrectly reformatted for the next.
Benchmarking the "Messy Reality"
Recent research has shifted from toy problems to high-fidelity benchmarks that simulate these complex workflows:
- Enterprise Workflows: WorkArena++, introduced by Léo Boisvert, shows that while humans achieve a 94% success rate on multi-step office tasks, state-of-the-art models like GPT-4o struggle to break 4%.
- Life-Serving Tasks: VitaBench, developed by Yueqing Sun, tests agents on coordinating across domains like food delivery and travel planning. Even advanced models like OpenAI’s o3 achieve only a 30% success rate when forced to manage interdependent constraints across these domains.
- The Model Context Protocol (MCP): To standardize how agents talk to tools, Zhenting Wang and others utilized the Model Context Protocol in MCP-Bench. They found that even with a unified protocol, agents struggle with the long-horizon planning required to use the output of one domain-specific tool as the input for another.
Emerging Architectures for Multi-App Control
To bridge this gap, researchers are moving away from monolithic "single-agent" designs toward hierarchical and specialized frameworks:
1. The Steward-Staff Model In MobileSteward, Yuxuan Liu proposes a centralized "StewardAgent" that handles high-level meta-planning, delegating specific UI interactions to specialized "StaffAgents." This separation of concerns prevents the high-level logic from being "muddied" by the low-level noise of UI navigation.
2. Dual-Agent Windows Interaction Similarly, the UFO framework by Chaoyun Zhang uses a "HostAgent" to observe the entire desktop and an "AppAgent" to focus on specific software. This allows the system to maintain a global state of the user's goal while executing local actions within apps like PowerPoint or Outlook.
3. Procedural Meta-Reasoning Yao Fu and colleagues introduced PAMAR, which represents tasks as natural-language procedures. By explicitly reasoning about the steps of a cross-app workflow, the agent can better detect when its internal knowledge is insufficient and trigger external searches to bridge information gaps.
Why It Matters
The shift toward cross-application orchestration represents the transition of AI from a "chatbot" to a "digital coworker." As Lingpeng Kong demonstrates with OSWorld, the "execution gap" is the primary hurdle. Solving this requires more than just larger models; it requires robust state management, better memory of user intent across devices (as argued by Chirag Shah in Task Intelligence), and the ability to navigate the "messy" transitions between the heterogeneous tools we use every day.