Talk to Right Specialists: Routing and Planning in Multi-agent System for Question Answering

Feijie Wu, Zitao Li, Fei Wei, Yaliang Li, Bolin Ding, Jing GaoarXiv 2025 · DOI · arXiv

Introduction

Retrieval-Augmented Generation (RAG) has become the gold standard for grounding Large Language Models (LLMs) in external facts, effectively mitigating the risk of hallucinations. However, traditional RAG systems typically rely on a single, domain-specific database. This architecture struggles with complex, cross-domain queries that require synthesis across multiple specialized fields.

A common workaround is to merge disparate databases into a single, massive vector store. This approach introduces two significant challenges:

  1. Computational Overhead: Searching a colossal unified database is expensive and increases the likelihood of retrieving irrelevant context.
  2. Data Sovereignty: In sectors like healthcare, law, or finance, data is often siloed due to legal or competitive constraints. Organizations cannot simply pool their proprietary data into a centralized repository.

To address these issues, the authors propose RopMura, a multi-agent system designed to enable collaboration between specialized, siloed RAG agents without requiring them to share raw data.

The RopMura Framework

RopMura employs two primary mechanisms to handle varying query complexities while maintaining data privacy.

1. Privacy-Preserving Routing

For single-hop queries, RopMura routes questions to the most relevant agent based on "knowledge boundaries." Instead of sharing raw data, each agent clusters its internal knowledge embeddings and shares only the cluster centroids with a central router.

Overview of the RopMura routing mechanism for single-hop queries.

The router compares the user's query embedding against these centroids to select the top-$k$ agents. These agents then locally generate an analysis and an answer. The router evaluates these responses for logical consistency and reliability before curating the final output. This "delegated generation" keeps the central context window clean and preserves the privacy of the underlying documents.

2. Dynamic Planning for Multi-Hop Reasoning

Complex questions requiring information from multiple domains are handled by a fine-grained greedy planner. Unlike static planners that break down a question once at the start, RopMura’s planner is iterative and adaptive.

The dynamic planning workflow for resolving multi-hop queries.

The process involves four key submodules:

  • Splitter: Decomposes the complex query into manageable sub-questions.
  • Selector: Identifies the optimal sub-question to address first.
  • Judger: Determines if the gathered information is sufficient to answer the original query. If not, it triggers the Splitter to generate new sub-questions based on the updated context.
  • Defender: Compiles the final, comprehensive response once all necessary information is retrieved.

Evaluation and Results

The system was tested across three diverse datasets: Natural Questions (single-hop), HotpotQA (multi-hop reasoning), and Multi-hop RAG (cross-domain news articles).

  • Single-Hop Performance: On Natural Questions, RopMura achieved a 78.26% GPT-evaluated accuracy, significantly outperforming single-agent baselines like ReAct (68.39%).
  • Multi-Hop Performance: The system excelled on complex tasks, reaching 85.03% accuracy on the Multi-hop RAG dataset. On HotpotQA, it achieved 74.93%, surpassing static planning methods like "One-shot" (73.75%) and "Presplit" (73.00%).
  • Operational Cost: The high accuracy of dynamic planning comes with a trade-off in efficiency. For HotpotQA, RopMura consumed approximately 113,000 tokens per query, highlighting the computational intensity of iterative multi-agent coordination.

Key Insights and Conclusion

RopMura demonstrates that using cluster centroids as privacy proxies is a scalable and effective method for routing queries in a multi-agent environment. By allowing agents to perform local reasoning (analysis + answer) rather than just returning raw text, the system offloads the cognitive burden from the central router and improves overall accuracy.

While the system is currently token-intensive, it provides a robust blueprint for cross-domain QA that respects data sovereignty. Future work aims to extend this framework to multimodal agents, enabling text queries to be matched against visual or audio knowledge boundaries.

Summary updated

Abstract

Retrieval-augmented generation (RAG) agents are increasingly deployed to answer questions over local knowledge bases that cannot be centralized due to knowledge-sovereignty constraints. This results in two recurring failures in production: users do not know which agent to consult, and complex questions require evidence distributed across multiple agents. To overcome these challenges, we propose RIRS, a training-free orchestration framework to enable a multi-agent system for question answering. In detail, RIRS summarizes each agent's local corpus in an embedding space, enabling a user-facing server to route queries only to the most relevant agents, reducing latency and avoiding noisy "broadcast-to-all" contexts. For complicated questions, the server can iteratively aggregate responses to derive intermediate results and refine the question to bridge the gap toward a comprehensive answer. Extensive experiments demonstrate the effectiveness of RIRS, including its ability to precisely select agents and provide accurate responses to single-hop queries, and its use of an iterative strategy to achieve accurate, multi-step resolutions for complex queries.
BibTeX
@misc{wu2025talk,
  author = {Feijie Wu and Zitao Li and Fei Wei and Yaliang Li and Bolin Ding and Jing Gao},
  title = {{Talk to Right Specialists: Routing and Planning in Multi-agent System for Question Answering}},
  year = {2025},
  eprint = {2501.07813},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.07813},
}
Figure