AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant

Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, Zhiyong Wu

Abstract

Current autonomous computer assistants face a trade-off between the broad reasoning of generalist Multimodal Large Language Models (MLLMs) and the precise execution of domain-specific specialized agents. We present AgentStore, a scalable framework that dynamically integrates diverse specialized agents into a unified "MetaAgent." By representing agents as learnable "AgentTokens," AgentStore avoids the context-window limitations of traditional in-context learning. Our approach achieves a 23.85% success rate on the OSWorld benchmark, more than doubling previous state-of-the-art results, and demonstrates robust performance in complex, multi-agent collaborative tasks.

Introduction

The quest for a truly autonomous computer assistant has long been hindered by the "Generalist-Specialist" dilemma. General-purpose MLLMs excel at high-level reasoning but often lack the granular, programmatic knowledge required for specialized software like professional spreadsheet tools. Conversely, specialized agents are highly proficient within their specific domains but fail when tasks require cross-application coordination. AgentStore addresses this by creating a scalable ecosystem where a central MetaAgent coordinates a diverse pool of specialists.

The AgentStore Framework

AgentStore operates on a modular philosophy, consisting of a standardized enrollment protocol and a novel token-based integration method.

AgentEnroll and the AgentPool

To ensure scalability, the authors developed AgentEnroll, a protocol that allows developers to register new specialized agents into an AgentPool. Each agent is accompanied by a descriptive document outlining its supported applications, specific capabilities, and inherent limitations.

The AgentEnroll protocol standardizes how specialized agents are added to the AgentPool with clear definitions of capabilities and limitations.

Integration via AgentTokens

A key technical innovation is the use of AgentTokens. Rather than relying on lengthy text descriptions in a prompt, AgentStore represents each specialized agent as a unique, learnable embedding in the MetaAgent's vocabulary. When the MetaAgent predicts a specific <Agent_i> token, it triggers the corresponding specialist. This method is highly efficient; training a new AgentToken takes approximately 0.2 hours on an A100, significantly faster than traditional fine-tuning methods.

Task Coordination: The MetaAgent as Hash Manager

For complex, open-ended tasks, the MetaAgent functions as a "Hash Manager." It decomposes a user's request into manageable sub-tasks and identifies the most relevant specialists from the pool.

AgentStore coordinates specialized agents for both single-domain tasks and cross-domain collaborative tasks requiring message passing between agents.

In collaborative scenarios, the MetaAgent manages message passing between agents. For instance, a "WebAgent" might extract metadata from a research site, which is then passed to a "SlideAgent" to be formatted into a presentation. This hierarchical approach allows the system to handle tasks that are impossible for any single specialist to complete in isolation.

Experimental Evaluation

The researchers evaluated AgentStore across several rigorous benchmarks:

  • OSWorld: AgentStore achieved a 23.85% success rate, a dramatic improvement over the previous state-of-the-art (11.21%).
  • APPAgent: In mobile environments, the system increased the success rate from 26.7% (GPT-4o baseline) to 57.8%.
  • OSWorld-Multi: In this new benchmark designed for collaboration, AgentStore's token-based management significantly outperformed standard In-Context Learning (ICL) baselines in correctly identifying and sequencing the necessary agents.

Conclusion

AgentStore demonstrates that the path to a generalist computer assistant lies in creating a scalable ecosystem of specialists. By treating agents as modular resources integrated through efficient learnable embeddings, AgentStore provides a robust blueprint for the next generation of autonomous AI assistants.

Summary updated

BibTeX
@inproceedings{jia2025agentstore,
  author = {Chengyou Jia and Minnan Luo and Zhuohang Dang and Qiushi Sun and Fangzhi Xu and Junlin Hu and Tianbao Xie and Zhiyong Wu},
  title = {{AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure