Agentic Meta-Orchestrator for Multi-Task Copilots

Xiaofeng Zhu, Yi Zhou2025 · DOI

TL;DR

This paper introduces the Agentic Meta-Orchestrator (AMO), a novel system designed to efficiently manage and route user requests to a growing number of specialized AI agents within Microsoft Copilot. AMO leverages meta-learning to dynamically select the optimal inference strategy, ensuring scalability and improved performance for complex, multi-task copilots.

What Problem Does This Paper Address?

Modern AI copilots, like Microsoft Copilot, are becoming increasingly sophisticated, integrating numerous specialized "agents" capable of handling diverse tasks—from e-commerce inquiries to code compliance checks. The challenge lies in effectively orchestrating these agents. As the number of agents grows, traditional classification methods struggle with scalability and handling overlapping agent functionalities. Furthermore, determining the best sequence of agent interactions for a given user prompt (inference planning) is complex. This paper addresses the need for a robust, scalable, and dynamic orchestrator that can seamlessly integrate new agents and intelligently plan agent interactions to provide accurate and context-aware responses.

Methods

The core of the proposed solution is the Agentic Meta-Orchestrator (AMO). AMO tackles the orchestration problem in three key ways:

  1. Hierarchical Text Classification to Multi-Level Learning-to-Rank: Instead of traditional multi-class classification, AMO reframes the task as a multi-level relevance rating learning-to-rank problem. Agent descriptions are treated as "candidate documents," and a model learns to rank them based on their relevance to the user prompt. This approach is more scalable and handles semantic overlaps better than standard classification.
  2. Memory-Efficient Inference with LoRA Arms: To manage the memory overhead of multiple large language models (LLMs), AMO utilizes LoRA (Low-Rank Adaptation) "arms." Each LoRA arm is a fine-tuned adapter for a specific task, allowing multiple task-specific models to share the same base LLM memory during inference, thus improving scalability and flexibility.
  3. Meta-Learning for Inference Planning: AMO employs a meta-learning decision tree model to determine the optimal inference strategy. This model learns to decide the best combination and order of agents for a given prompt, going beyond simple sequential execution or predefined workflows. It can even allow for revisiting nodes, enabling retries and more robust planning.

Datasets

The paper mentions using two production use cases for evaluation:

  • M365 E-Commerce Copilot: This dataset consists of approximately 30,000 user prompts with labeled agent classes such as "Ask for Price," "Non-Microsoft Products," "Contact Human Support Agents," and product entities.
  • Code Compliance Copilot: This dataset involves internal DevOps pull requests, with hierarchical classes like "GB issue," "non-GB issue," "GB display issue," and "GB encoding issue."
  • Amazon Product Review Dataset: For evaluating the hierarchical text classification approach, a three-level hierarchical dataset from Amazon product reviews was used, containing 40,000 training and 10,000 validation records.

Key Insights and Ideas

  • Learning-to-Rank for Agent Orchestration: Reframing agent selection as a learning-to-rank problem offers significant advantages in scalability and flexibility compared to traditional classification, especially as the number of agents grows.
  • LoRA Arms for Efficient Multi-Tasking: Sharing a base LLM with task-specific LoRA adapters is a practical approach to reduce memory footprint while enabling efficient inference for multiple agents.
  • Meta-Learning for Dynamic Planning: A meta-learning decision tree can effectively learn complex inference strategies, allowing the copilot to adapt its agent interaction sequence based on the specific user prompt and context.
  • Hierarchical Text Classification as Multi-Level Rating: Treating hierarchical classification as a multi-level rating task, rather than a generation task, leads to more predictable and controllable outputs, which is crucial for production systems.

Key Results

The paper demonstrates the effectiveness of AMO through several experiments:

  • Scalability with Growing Agents: Figure 5 shows that AMO's classification performance remains stable as new agents are added, while traditional methods like BERT see a significant performance drop.
  • End-to-End Performance Gains: Table I highlights that AMO achieves superior end-to-end performance (measured by ROUGE-L, BERTScore, and F-1 measures) for both the M365 E-Commerce Copilot and Code Compliance Copilot compared to baselines like fine-tuned Phi-3.5, ChatGPT-40, and AutoGen. For instance, in the M365 E-Commerce Copilot, AMO shows a 15.93% F-1 score for Agent Orchestration Classification, while fine-tuned Phi-3.5 achieves 8.34% and ChatGPT-40 shows -13.67%.
  • Hierarchical Text Classification Accuracy: Table II shows that AMO achieves high accuracy (e.g., 0.96 for Level-1 Accuracy on the Amazon Product Review Dataset) in hierarchical text classification tasks, outperforming few-shot LLM approaches and supervised text classifiers.

Limitations and Future Work

  • Online Learning and Reinforcement Learning: The authors acknowledge that they did not focus on incorporating online learning leveraging reinforcement learning for the current production release. They suggest this as an interesting area for future work to enable more adaptive and consistent agent actions.
  • Multi-Modal Inputs: The paper mentions that additional signals, such as user interaction, could be leveraged for multi-modal inputs, which is a potential avenue for future research.
  • Federated Learning and Collaborative Intelligence: The authors plan to explore federated learning and other collaborative intelligence approaches for future copilot development.

Summary updated

Abstract

Microsoft Copilot suites serve as the universal entry point for various agents skilled in handling important tasks, ranging from assisting a customer with product purchases to detecting vulnerabilities in corporate programming code. Each agent can be powered by language models, software engineering operations, such as database retrieval, and internal & external knowledge. The repertoire of a copilot can expand dynamically with new agents. This requires a robust orchestrator that can distribute tasks from user prompts to the right agents. In this work, we propose an Agentic Meta-orchestrator (AMO) for handling multiple tasks and scalable agents in copilot services, which can provide both natural language and action responses. We will also demonstrate the planning that leverages meta- learning, i.e., a trained decision tree model for deciding the best inference strategy among various agents/models. We showcase the effectiveness of our AMO through two production use cases: Microsoft 365 (M365) E-Commerce Copilot and code compliance copilot. M365 E-Commerce Copilot advertises Microsoft products to external customers to promote sales success. The M365 E- Commerce Copilot provides up-to-date product information and connects to multiple agents, such as relational databases and human customer support. The code compliance copilot scans the internal DevOps code to detect known and new compliance issues in pull requests (PR).
BibTeX
@misc{zhu2025agentic,
  author = {Xiaofeng Zhu and Yi Zhou},
  title = {{Agentic Meta-Orchestrator for Multi-Task Copilots}},
  year = {2025},
  doi = {10.1109/icdmw69685.2025.00206},
}
Figure