PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answering

Yu Liu, Wenxiao Zhang, Cong Cao, Wenxuan Lu, Fangfang Yuan, Diandian Guo, Kun Peng, Qiang Sun, Kaiyan Zhang, Yanbing Liu, Jin B. Hong, Bowen Zhou, Zhiyuan MaarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces PRISMA, a novel multi-agent system that tackles the complex challenge of open-domain, multi-hop question answering. By decoupling planning, retrieval, and reasoning into specialized agents that collaborate through an "inspect-and-correct" feedback loop, PRISMA significantly improves accuracy and robustness, achieving state-of-the-art results on challenging benchmarks.

What Problem Does This Paper Address?

Answering complex questions that require piecing together information from multiple documents (multi-hop question answering) is a major hurdle for AI systems, especially when dealing with vast amounts of text (open-domain). Existing methods, often based on Retrieval-Augmented Generation (RAG), struggle with two key issues:

  1. Retrieval Collapse: When systems repeatedly search for information without a clear plan, they can fail to find crucial "bridge" answers needed to connect different pieces of information, leading to reasoning failures.
  2. Learning Instability: Training these complex systems end-to-end is difficult. It's hard to assign credit for success or pinpoint errors across different modules, making them prone to overfitting and lacking robustness.

PRISMA aims to solve these problems by creating a more structured and collaborative approach to multi-hop QA.

Methods

PRISMA employs a unique Plan-Retrieve-Inspect-Solve-Memoize (PRISM) architecture, inspired by how human researchers tackle complex problems.

  • Planner: Breaks down a complex question into a series of smaller, dependency-aware subquestions.
  • Retriever: Fetches relevant documents for each subquestion using a sophisticated three-stage cascade (dense, hybrid, and cross-encoder).
  • Inspector: This is the core innovation. It acts as a quality control agent, checking both the subquestion's clarity and the retrieved documents' relevance. If issues are found, it triggers targeted repairs, such as rewriting the subquestion or expanding the search.
  • Solver: Uses the retrieved evidence to generate an answer for each subquestion, ensuring it's grounded in the provided text.
  • Memoizer: Caches previously answered subquestions to speed up future queries.

The system is trained using a Two-Stage Group Relative Policy Optimization (GRPO) approach:

  • Stage I: Calibrates the Planner and Solver independently to excel at their specific tasks.
  • Stage II: Trains the Inspector using Observation-Aware Residual Policy Optimization (OARPO). This stage focuses on teaching the Inspector to identify and correct errors made by the already-trained Planner and Solver, using expert trajectories as a guide. This "residual audit" approach helps stabilize training and improve overall system reliability.

Datasets

PRISMA was evaluated on a comprehensive set of datasets:

  • In-distribution (used for training): MuSiQue, HotpotQA, and 2WikiMultiHopQA.
  • Out-of-distribution (for generalization testing): NaturalQuestions (NQ), Bamboogle, and five newly constructed domain-specific datasets (Chemistry, Food, Game, Geography, Music). These custom datasets were designed to test cross-domain transfer without relying solely on Wikipedia-centric knowledge.

Key Insights and Ideas

  • The Power of Collaboration and Inspection: The most striking insight is the critical role of the Inspector agent. It's not just a passive checker; its ability to detect errors and trigger targeted recovery (like re-retrieval or subquestion rewriting) is crucial for overcoming retrieval failures and improving overall accuracy.
  • Decoupling for Stability: Breaking down the complex QA pipeline into specialized agents and training them in stages (expert calibration followed by residual audit) significantly improves learning stability compared to end-to-end training.
  • Human-like Workflow: Mimicking the human research process of planning, consulting, reasoning, and verifying proves to be a highly effective strategy for tackling complex QA.
  • Retrieval is Still Key, But Needs Guidance: While advanced retrieval methods are used, the paper highlights that retrieval alone isn't enough. It needs to be guided by a plan and validated by an inspector to avoid "retrieval collapse."

Key Results

  • State-of-the-Art Performance: PRISMA achieved state-of-the-art results on ten benchmarks, outperforming existing trained RAG systems and even API-based models on challenging multi-hop QA tasks.
  • Significant Gains on Harder Questions: PRISMA showed particularly strong performance on questions requiring more hops (e.g., 3-hop and 4-hop questions on MuSiQue), where it significantly outperformed baselines. For instance, on 3-hop questions, PRISMA achieved 23.6% EM compared to 8.9% for a strong baseline.
  • Robustness and Generalization: The system demonstrated strong generalization capabilities on out-of-distribution datasets, indicating its ability to handle diverse domains and reasoning types.
  • Efficiency Gains: The Memoizer component provided a significant speedup (29% on average) with minimal impact on accuracy, making PRISMA more practical for real-world deployment.
  • Inspector's Impact: Ablation studies showed that removing the Inspector agent led to a catastrophic performance drop, highlighting its central role in the system's success.

Limitations and Future Work

  • Resource Intensive: PRISMA requires substantial memory and GPU resources for deployment due to its complex architecture and the need to manage a large knowledge base.
  • Training Complexity: Optimizing reward functions for individual agents and ensuring proper alignment across modules remains a delicate task.
  • API Model Competition: The rapid advancement of large API-based LLMs presents an ongoing challenge, and future work may require even more compact foundational models to truly surpass them across all domains.
  • Subquestion Disambiguation: The paper acknowledges that ambiguous subquestions (e.g., "main character") can still lead to unrecoverable errors, suggesting future work on improving the Planner's ability to disambiguate.

Summary updated

Abstract

Answering real-world open-domain multi-hop questions over massive corpora is a critical challenge in Retrieval-Augmented Generation (RAG) systems. Recent research employs reinforcement learning (RL) to end-to-end optimize the retrieval-augmented reasoning process, directly enhancing its capacity to resolve complex queries. However, reliable deployment is hindered by two obstacles. 1) Retrieval Collapse: iterative retrieval over large corpora fails to locate intermediate evidence containing bridge answers without reasoning-guided planning, causing downstream reasoning to collapse. 2) Learning Instability: end-to-end trajectory training suffers from weak credit assignment across reasoning chains and poor error localization across modules, causing overfitting to benchmark-specific heuristics that limit transferability and stability. To address these problems, we propose PRISMA, a decoupled RL-guided framework featuring a Plan-Retrieve-Inspect-Solve-Memoize architecture. PRISMA's strength lies in reasoning-guided collaboration: the Inspector provides reasoning-based feedback to refine the Planner's decomposition and fine-grained retrieval, while enforcing evidence-grounded reasoning in the Solver. We optimize individual agent capabilities via Two-Stage Group Relative Policy Optimization (GRPO). Stage I calibrates the Planner and Solver as specialized experts in planning and reasoning, while Stage II utilizes Observation-Aware Residual Policy Optimization (OARPO) to enhance the Inspector's ability to verify context and trigger targeted recovery. Experiments show that PRISMA achieves state-of-the-art performance on ten benchmarks and can be deployed efficiently in real-world scenarios.
BibTeX
@misc{liu2026prisma,
  author = {Yu Liu and Wenxiao Zhang and Cong Cao and Wenxuan Lu and Fangfang Yuan and Diandian Guo and Kun Peng and Qiang Sun and Kaiyan Zhang and Yanbing Liu and Jin B. Hong and Bowen Zhou and Zhiyuan Ma},
  title = {{PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answering}},
  year = {2026},
  eprint = {2601.05465},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.05465},
}
Figure