Abstract
Multi-agent Large Language Model (LLM) systems, typically composed of a "meta-thinking" planner and a "reasoning" executor, promise enhanced problem-solving capabilities. However, these systems often fall victim to "lazy agent behavior," where the system collapses into a single-agent setup as one agent fails to contribute meaningful value. This paper identifies that standard Reinforcement Learning (RL) objectives, such as Group Relative Preference Optimization (GRPO), mathematically incentivize this laziness by penalizing longer trajectories. We introduce Dr. MAMR (Multi-Agent Meta-Reasoning Done Right), a framework that employs a Shapley-inspired causal influence metric and a verifiable restart mechanism to enforce collaboration. Our results demonstrate significant performance gains across complex mathematical benchmarks, achieving a 20.00% accuracy on AIME24 compared to 16.67% for single-agent baselines.
The Challenge of Lazy Agents
Recent advancements in multi-agent reasoning have encountered a persistent failure mode: the reasoning agent often outputs empty strings or merely echoes the planner's instructions. This "laziness" is not a lack of capability but a rational response to flawed optimization. In standard multi-turn GRPO, the loss function normalizes rewards by the number of turns. This creates a mathematical bias toward shorter interactions, teaching agents that the most efficient path to a reward is to terminate the conversation as quickly as possible.
Furthermore, in extended reasoning tasks, agents frequently suffer from "sunk cost" fallacies. Once a mistake is made in the early turns, the agent often spends subsequent turns attempting to justify the error rather than correcting it, leading to a total failure of the reasoning chain.
The Dr. MAMR Framework
To address these issues, Dr. MAMR introduces three core innovations designed to foster genuine deliberation and robust credit assignment.
1. Normalization Debias
We eliminate the turn-level normalization in the GRPO objective. By removing the penalty for longer trajectories, we allow the model to explore more complex, multi-turn reasoning paths without being prematurely incentivized to finish.
2. Shapley-inspired Causal Influence (CI)
To ensure both agents contribute, we implement a Causal Influence metric. This metric measures the impact of each reasoning step on the final outcome. To make this computationally feasible for online RL, we group semantically similar steps across rollouts and calculate their average impact on the model's next-step logits. This ensures that agents are rewarded for the utility of their ideas rather than just their presence in the transcript.
3. Verifiable Restart Mechanism
We introduce a <restart> control token that allows the reasoning agent to discard its current history if it detects a mistake. This mechanism is trained using a "verifiable reward": the agent receives a positive signal if invoking a restart leads to a higher confidence in the correct final answer. This mimics human-like deliberation, where one might "start over" upon realizing a fundamental error in logic.
Empirical Analysis of Agent Contribution
The effectiveness of the Causal Influence (CI) metric is evident when examining the distribution of influence across different training regimes.
As shown in the figure above, standard multi-agent approaches (like ReMA) often result in overlapping or poorly defined influence distributions, indicating that the roles of the planner and executor are blurred or dominated by one party. In contrast, Dr. MAMR ("Ours") produces a distinct separation and specific distribution of influence, confirming that the framework successfully assigns credit and forces both agents to play unique, valuable roles in the reasoning process.
Key Results and Benchmarks
Dr. MAMR was evaluated against eight rigorous benchmarks, including MATH500, AIME 24/25, and OlympiadBench.
- Superior Performance: On the AIME24 benchmark, Dr. MAMR (using a Qwen2.5-7B base) reached 20.00% accuracy, significantly outperforming the previous multi-agent state-of-the-art, ReMA (13.33%), and single-agent GRPO (16.67%).
- Scaling with Complexity: The performance gap between Dr. MAMR and other methods increased as problem difficulty rose, suggesting that the deliberation and restart mechanisms are particularly vital for high-level competition math.
- Training Stability: Unlike standard multi-agent RL, which often collapses on smaller (3B) models, Dr. MAMR maintained stable progress, proving that proper credit assignment is key to scaling multi-agent systems.
Conclusion
Dr. MAMR demonstrates that multi-agent reasoning failures are often rooted in optimization biases rather than model size alone. By debiasing the RL objective and introducing explicit metrics for causal contribution and deliberation, we can unlock the true potential of collaborative LLM systems. Future work will explore applying these principles to non-mathematical domains such as code generation and long-form strategic planning.
Summary updated