Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation

Zhiwei Zhang, Xiaomin Li, Yudi Lin, Hui Liu, Ramraj Chandradevan, Linlin Wu, Minhua Lin, Fali Wang, Xianfeng Tang, Qi He, Suhang Wang

Abstract

Multi-agent Large Language Model (LLM) systems, typically composed of a "meta-thinking" planner and a "reasoning" executor, promise enhanced problem-solving capabilities. However, these systems often fall victim to "lazy agent behavior," where the system collapses into a single-agent setup as one agent fails to contribute meaningful value. This paper identifies that standard Reinforcement Learning (RL) objectives, such as Group Relative Preference Optimization (GRPO), mathematically incentivize this laziness by penalizing longer trajectories. We introduce Dr. MAMR (Multi-Agent Meta-Reasoning Done Right), a framework that employs a Shapley-inspired causal influence metric and a verifiable restart mechanism to enforce collaboration. Our results demonstrate significant performance gains across complex mathematical benchmarks, achieving a 20.00% accuracy on AIME24 compared to 16.67% for single-agent baselines.

The Challenge of Lazy Agents

Recent advancements in multi-agent reasoning have encountered a persistent failure mode: the reasoning agent often outputs empty strings or merely echoes the planner's instructions. This "laziness" is not a lack of capability but a rational response to flawed optimization. In standard multi-turn GRPO, the loss function normalizes rewards by the number of turns. This creates a mathematical bias toward shorter interactions, teaching agents that the most efficient path to a reward is to terminate the conversation as quickly as possible.

Furthermore, in extended reasoning tasks, agents frequently suffer from "sunk cost" fallacies. Once a mistake is made in the early turns, the agent often spends subsequent turns attempting to justify the error rather than correcting it, leading to a total failure of the reasoning chain.

The Dr. MAMR Framework

To address these issues, Dr. MAMR introduces three core innovations designed to foster genuine deliberation and robust credit assignment.

Conceptual overview of lazy vs. non-lazy reasoning and the Dr. MAMR mechanisms for causal influence and verifiable restarts.

1. Normalization Debias

We eliminate the turn-level normalization in the GRPO objective. By removing the penalty for longer trajectories, we allow the model to explore more complex, multi-turn reasoning paths without being prematurely incentivized to finish.

2. Shapley-inspired Causal Influence (CI)

To ensure both agents contribute, we implement a Causal Influence metric. This metric measures the impact of each reasoning step on the final outcome. To make this computationally feasible for online RL, we group semantically similar steps across rollouts and calculate their average impact on the model's next-step logits. This ensures that agents are rewarded for the utility of their ideas rather than just their presence in the transcript.

3. Verifiable Restart Mechanism

We introduce a <restart> control token that allows the reasoning agent to discard its current history if it detects a mistake. This mechanism is trained using a "verifiable reward": the agent receives a positive signal if invoking a restart leads to a higher confidence in the correct final answer. This mimics human-like deliberation, where one might "start over" upon realizing a fundamental error in logic.

Empirical Analysis of Agent Contribution

The effectiveness of the Causal Influence (CI) metric is evident when examining the distribution of influence across different training regimes.

Comparison of Causal Influence distributions between Meta-thinking and Reasoning agents across different training methods.

As shown in the figure above, standard multi-agent approaches (like ReMA) often result in overlapping or poorly defined influence distributions, indicating that the roles of the planner and executor are blurred or dominated by one party. In contrast, Dr. MAMR ("Ours") produces a distinct separation and specific distribution of influence, confirming that the framework successfully assigns credit and forces both agents to play unique, valuable roles in the reasoning process.

Key Results and Benchmarks

Dr. MAMR was evaluated against eight rigorous benchmarks, including MATH500, AIME 24/25, and OlympiadBench.

  • Superior Performance: On the AIME24 benchmark, Dr. MAMR (using a Qwen2.5-7B base) reached 20.00% accuracy, significantly outperforming the previous multi-agent state-of-the-art, ReMA (13.33%), and single-agent GRPO (16.67%).
  • Scaling with Complexity: The performance gap between Dr. MAMR and other methods increased as problem difficulty rose, suggesting that the deliberation and restart mechanisms are particularly vital for high-level competition math.
  • Training Stability: Unlike standard multi-agent RL, which often collapses on smaller (3B) models, Dr. MAMR maintained stable progress, proving that proper credit assignment is key to scaling multi-agent systems.

Conclusion

Dr. MAMR demonstrates that multi-agent reasoning failures are often rooted in optimization biases rather than model size alone. By debiasing the RL objective and introducing explicit metrics for causal contribution and deliberation, we can unlock the true potential of collaborative LLM systems. Future work will explore applying these principles to non-mathematical domains such as code generation and long-form strategic planning.

Summary updated

BibTeX
@inproceedings{zhang2026unlocking,
  author = {Zhiwei Zhang and Xiaomin Li and Yudi Lin and Hui Liu and Ramraj Chandradevan and Linlin Wu and Minhua Lin and Fali Wang and Xianfeng Tang and Qi He and Suhang Wang},
  title = {{Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=5J6u03ObRZ},
}
Figure