Abstract
Current autonomous computer assistants face a trade-off between the broad reasoning of generalist Multimodal Large Language Models (MLLMs) and the precise execution of domain-specific specialized agents. We present AgentStore, a scalable framework that dynamically integrates diverse specialized agents into a unified "MetaAgent." By representing agents as learnable "AgentTokens," AgentStore avoids the context-window limitations of traditional in-context learning. Our approach achieves a 23.85% success rate on the OSWorld benchmark, more than doubling previous state-of-the-art results, and demonstrates robust performance in complex, multi-agent collaborative tasks.
Introduction
The quest for a truly autonomous computer assistant has long been hindered by the "Generalist-Specialist" dilemma. General-purpose MLLMs excel at high-level reasoning but often lack the granular, programmatic knowledge required for specialized software like professional spreadsheet tools. Conversely, specialized agents are highly proficient within their specific domains but fail when tasks require cross-application coordination. AgentStore addresses this by creating a scalable ecosystem where a central MetaAgent coordinates a diverse pool of specialists.
The AgentStore Framework
AgentStore operates on a modular philosophy, consisting of a standardized enrollment protocol and a novel token-based integration method.
AgentEnroll and the AgentPool
To ensure scalability, the authors developed AgentEnroll, a protocol that allows developers to register new specialized agents into an AgentPool. Each agent is accompanied by a descriptive document outlining its supported applications, specific capabilities, and inherent limitations.
Integration via AgentTokens
A key technical innovation is the use of AgentTokens. Rather than relying on lengthy text descriptions in a prompt, AgentStore represents each specialized agent as a unique, learnable embedding in the MetaAgent's vocabulary. When the MetaAgent predicts a specific <Agent_i> token, it triggers the corresponding specialist. This method is highly efficient; training a new AgentToken takes approximately 0.2 hours on an A100, significantly faster than traditional fine-tuning methods.
Task Coordination: The MetaAgent as Hash Manager
For complex, open-ended tasks, the MetaAgent functions as a "Hash Manager." It decomposes a user's request into manageable sub-tasks and identifies the most relevant specialists from the pool.
In collaborative scenarios, the MetaAgent manages message passing between agents. For instance, a "WebAgent" might extract metadata from a research site, which is then passed to a "SlideAgent" to be formatted into a presentation. This hierarchical approach allows the system to handle tasks that are impossible for any single specialist to complete in isolation.
Experimental Evaluation
The researchers evaluated AgentStore across several rigorous benchmarks:
- OSWorld: AgentStore achieved a 23.85% success rate, a dramatic improvement over the previous state-of-the-art (11.21%).
- APPAgent: In mobile environments, the system increased the success rate from 26.7% (GPT-4o baseline) to 57.8%.
- OSWorld-Multi: In this new benchmark designed for collaboration, AgentStore's token-based management significantly outperformed standard In-Context Learning (ICL) baselines in correctly identifying and sequencing the necessary agents.
Conclusion
AgentStore demonstrates that the path to a generalist computer assistant lies in creating a scalable ecosystem of specialists. By treating agents as modular resources integrated through efficient learnable embeddings, AgentStore provides a robust blueprint for the next generation of autonomous AI assistants.
Summary updated