TL;DR
This paper introduces MATTRL, a novel framework that enhances multi-agent reasoning by injecting structured textual experiences during inference. By forming expert teams, retrieving relevant past interactions, and using a refined credit assignment mechanism, MATTRL significantly improves accuracy on complex tasks in medicine, math, and education, offering a stable and efficient path to robust multi-agent reasoning.
What Problem Does This Paper Address?
The paper tackles the challenges of training multi-agent reinforcement learning (MARL) systems, which are notoriously resource-intensive and unstable. Key issues include non-stationarity caused by co-adapting teammates and sparse, high-variance rewards. These problems make it difficult to achieve reliable and robust multi-agent reasoning, especially when dealing with distribution shifts (i.e., new or unseen tasks/domains). Existing methods often require extensive training or struggle with stability, limiting their practical application.
Methods
MATTRL introduces a test-time adaptation framework that leverages structured textual experience during inference, rather than requiring additional training. The core components are:
- Multi-Expert Team Collaboration: A coordinator agent forms a team of specialist agents from a predefined pool, tailored to the specific task.
- Experience-Augmented Dialogue: Specialists engage in multi-turn discussions, where they retrieve and integrate relevant textual experiences from a shared pool to inform their opinions. This process continues until consensus is reached or a turn limit is met.
- Credit Assignment: A crucial aspect is how to assign credit for successful collaboration. The paper explores various schemes (naive averaging, difference rewards, Shapley-style approximations) to identify high-value dialogue turns. These turns are then distilled into textual experiences.
- Test-Time Experience Retrieval: At inference, each specialist retrieves relevant experiences based on the current context and appends them to their prompts, guiding their reasoning without updating model weights.
Datasets
The framework is evaluated on three challenging benchmarks:
- Medicine: RareBench Task 4, focusing on differential diagnosis for rare diseases, involving 2,185 cases across 421 diseases.
- Math: HLE (Humanity's Last Exam), a benchmark of text-only math problems with 856 samples.
- Education: SuperGPQA, a dataset used to evaluate teaching interactions, with GPT-4o as the student and GPT-5 as the teacher.
Key Insights and Ideas
- Test-Time Adaptation is Powerful: Injecting structured experiences at inference time, rather than relying solely on pre-training or prompt engineering, offers a significant boost in performance and robustness, especially against distribution shifts.
- Structured Experience Matters: Simply providing more context (few-shot learning) is less effective than integrating carefully curated, actionable textual experiences that guide reasoning.
- Credit Assignment is Key for Experience Quality: The method used to assign credit for collaborative turns directly impacts the quality and usefulness of the extracted experiences. Difference Rewards emerged as a strong performer, balancing precision and efficiency.
- Team Size Optimization: Performance doesn't always scale linearly with team size. For tasks requiring strict precision (like Hit@1), smaller teams (around 3 agents) are often optimal, while larger teams might offer broader recall.
- Adaptive Routing: An adaptive router that intelligently selects between a single-agent approach and the MATTRL framework can further improve performance by matching the collaboration style to the complexity of the case.
Key Results
- Medicine: MATTRL achieved an average accuracy improvement of 3.67% over a multi-agent baseline and 8.67% over single-agent baselines. It outperformed other state-of-the-art methods like MDAgent and RareAgents-Refined in terms of retrieval quality.
- Math: MATTRL improved exact-match accuracy on HLE problems from 0.33 (multi-agent baseline) to 0.36, a significant gain over the single-agent baseline (0.27).
- Education: MATTRL achieved the best learning gains (∆Acc = 0.33), nearly doubling the improvement of the single-agent baseline, demonstrating its effectiveness in complex teaching scenarios.
- Credit Assignment: The "Difference Rewards" method for credit assignment showed the best strict-precision performance (Hit@1/3).
Limitations and Future Work
- Inference Compute and Latency: The method's inference-time compute and latency increase with multi-agent rollouts and exploration budgets.
- Experience Pool Drift: A growing test-time experience pool can become stale, duplicated, or spurious, potentially accumulating noise.
Future work will focus on:
- Introducing dynamic budget controllers and confidence-based early stopping to manage cost without sacrificing accuracy.
- Implementing lifecycle management for experiences (recency weighting, de-duplication, anomaly screening) to maintain precision over time.
Summary updated