Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning

Zhiyuan Hu, Yunhai Hu, Juncheng Liu, Shuyue Stella Li, Yucheng Wang, Zhen Xu, See-Kiong Ng, Anh Tuan Luu, Xinxing Xu, Bryan Hooi, Cynthia Breazeal, Hae Won ParkarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces MATTRL, a novel framework that enhances multi-agent reasoning by injecting structured textual experiences during inference. By forming expert teams, retrieving relevant past interactions, and using a refined credit assignment mechanism, MATTRL significantly improves accuracy on complex tasks in medicine, math, and education, offering a stable and efficient path to robust multi-agent reasoning.

What Problem Does This Paper Address?

The paper tackles the challenges of training multi-agent reinforcement learning (MARL) systems, which are notoriously resource-intensive and unstable. Key issues include non-stationarity caused by co-adapting teammates and sparse, high-variance rewards. These problems make it difficult to achieve reliable and robust multi-agent reasoning, especially when dealing with distribution shifts (i.e., new or unseen tasks/domains). Existing methods often require extensive training or struggle with stability, limiting their practical application.

Methods

MATTRL introduces a test-time adaptation framework that leverages structured textual experience during inference, rather than requiring additional training. The core components are:

  1. Multi-Expert Team Collaboration: A coordinator agent forms a team of specialist agents from a predefined pool, tailored to the specific task.
  2. Experience-Augmented Dialogue: Specialists engage in multi-turn discussions, where they retrieve and integrate relevant textual experiences from a shared pool to inform their opinions. This process continues until consensus is reached or a turn limit is met.
  3. Credit Assignment: A crucial aspect is how to assign credit for successful collaboration. The paper explores various schemes (naive averaging, difference rewards, Shapley-style approximations) to identify high-value dialogue turns. These turns are then distilled into textual experiences.
  4. Test-Time Experience Retrieval: At inference, each specialist retrieves relevant experiences based on the current context and appends them to their prompts, guiding their reasoning without updating model weights.

Datasets

The framework is evaluated on three challenging benchmarks:

  • Medicine: RareBench Task 4, focusing on differential diagnosis for rare diseases, involving 2,185 cases across 421 diseases.
  • Math: HLE (Humanity's Last Exam), a benchmark of text-only math problems with 856 samples.
  • Education: SuperGPQA, a dataset used to evaluate teaching interactions, with GPT-4o as the student and GPT-5 as the teacher.

Key Insights and Ideas

  • Test-Time Adaptation is Powerful: Injecting structured experiences at inference time, rather than relying solely on pre-training or prompt engineering, offers a significant boost in performance and robustness, especially against distribution shifts.
  • Structured Experience Matters: Simply providing more context (few-shot learning) is less effective than integrating carefully curated, actionable textual experiences that guide reasoning.
  • Credit Assignment is Key for Experience Quality: The method used to assign credit for collaborative turns directly impacts the quality and usefulness of the extracted experiences. Difference Rewards emerged as a strong performer, balancing precision and efficiency.
  • Team Size Optimization: Performance doesn't always scale linearly with team size. For tasks requiring strict precision (like Hit@1), smaller teams (around 3 agents) are often optimal, while larger teams might offer broader recall.
  • Adaptive Routing: An adaptive router that intelligently selects between a single-agent approach and the MATTRL framework can further improve performance by matching the collaboration style to the complexity of the case.

Key Results

  • Medicine: MATTRL achieved an average accuracy improvement of 3.67% over a multi-agent baseline and 8.67% over single-agent baselines. It outperformed other state-of-the-art methods like MDAgent and RareAgents-Refined in terms of retrieval quality.
  • Math: MATTRL improved exact-match accuracy on HLE problems from 0.33 (multi-agent baseline) to 0.36, a significant gain over the single-agent baseline (0.27).
  • Education: MATTRL achieved the best learning gains (∆Acc = 0.33), nearly doubling the improvement of the single-agent baseline, demonstrating its effectiveness in complex teaching scenarios.
  • Credit Assignment: The "Difference Rewards" method for credit assignment showed the best strict-precision performance (Hit@1/3).

Limitations and Future Work

  • Inference Compute and Latency: The method's inference-time compute and latency increase with multi-agent rollouts and exploration budgets.
  • Experience Pool Drift: A growing test-time experience pool can become stale, duplicated, or spurious, potentially accumulating noise.

Future work will focus on:

  • Introducing dynamic budget controllers and confidence-based early stopping to manage cost without sacrificing accuracy.
  • Implementing lifecycle management for experiences (recency weighting, de-duplication, anomaly screening) to maintain precision over time.

Summary updated

Abstract

Multi-agent systems have evolved into practical LLM-driven collaborators for many applications, gaining robustness from diversity and cross-checking. However, multi-agent RL (MARL) training is resource-intensive and unstable: co-adapting teammates induce non-stationarity, and rewards are often sparse and high-variance. Therefore, we introduce \textbf{Multi-Agent Test-Time Reinforcement Learning (MATTRL)}, a framework that injects structured textual experience into multi-agent deliberation at inference time. MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making. We also study credit assignment for constructing a turn-level experience pool, then reinjecting it into the dialogue. Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67\% over a multi-agent baseline, and by 8.67\% over comparable single-agent baselines. Ablation studies examine different credit-assignment schemes and provide a detailed comparison of how they affect training outcomes. MATTRL offers a stable, effective and efficient path to distribution-shift-robust multi-agent reasoning without tuning.
BibTeX
@misc{hu2026collaborative,
  author = {Zhiyuan Hu and Yunhai Hu and Juncheng Liu and Shuyue Stella Li and Yucheng Wang and Zhen Xu and See-Kiong Ng and Anh Tuan Luu and Xinxing Xu and Bryan Hooi and Cynthia Breazeal and Hae Won Park},
  title = {{Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning}},
  year = {2026},
  eprint = {2601.09667},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.09667},
}
Figure