Internalizing Self-Consistency in Language Models: Multi-Agent Consensus Alignment

Ankur Samanta, Akshayaa Magesh, Youliang Yu, Runzhe Wu, Ayush Jain, Daniel Jiang, Boris Vidolov, Paul Sajda, Yonathan Efroni, Kaveh Hassani

Abstract

Large Language Models (LLMs) often exhibit inconsistency in reasoning tasks, providing different answers to the same prompt across multiple trials. While inference-time techniques like majority voting can mitigate this, they do not improve the underlying model's intrinsic capabilities. This paper introduces Multi-Agent Consensus Alignment (MACA), a framework for self-improvement where models deliberate in a multi-agent debate to generate high-quality training signals. By applying preference learning to the resulting consensus, models internalize superior reasoning patterns. Experimental results across MATH, GSM8K, and other benchmarks show significant gains in both single-agent and multi-agent performance, often rivaling training on human-labeled data.

Introduction

Language models frequently produce varying answers to the same complex prompt. While techniques like majority voting or multi-agent debate can improve accuracy at inference time, these methods are computationally intensive and leave the base model's reasoning capabilities unchanged. The challenge of self-improvement lies in generating a training signal that surpasses the model's initial performance without relying on external human labels. The authors of this work ask: Can we use the collective consensus of a debate to create a superior training signal that helps a model internalize stable, consistent reasoning?

The MACA Framework

The researchers propose Multi-Agent Consensus Alignment (MACA), a three-stage pipeline designed to help models internalize the "wisdom of the crowd" generated through deliberation.

The MACA Framework overview showing the transition from debate to preference alignment.
  1. Multi-Agent Debate: Multiple instances (clones) of a base LLM engage in a multi-round debate. Each agent generates an initial reasoning path and answer, then iteratively updates its response after observing the reasoning of its peers.
  2. Consensus Partitioning: The final outputs are grouped based on the final majority. Reasoning traces that align with the majority consensus are designated as "winners," while dissenting traces are labeled as "losers."
  3. Alignment Training: The model undergoes post-training using these partitioned traces. The study explored several objectives, finding that preference-based methods like MV-DPO (Majority-Vote Direct Preference Optimization) and MV-KTO (Kahneman-Tversky Optimization) were most effective. These methods teach the model to prefer the reasoning steps that led to the correct consensus over those that did not.

Experimental Setup

The framework was evaluated using a diverse set of base models, including Qwen-2B, Llama-3B, Phi-4B, and Llama-8B. Training focused on mathematical reasoning datasets such as MATH, GSM8K, and MathQA. To test the robustness of the internalized reasoning, the models were also evaluated on unseen generalization benchmarks like GPQA (graduate-level science) and CommonsenseQA.

Key Insights

  • Deliberation as a Superior Signal: Multi-round debate captures the dynamics of logical correction, providing a richer signal than static majority voting. It captures the process of reaching a consensus, not just the final tally.
  • Preference Learning for Credit Assignment: By comparing winning and losing traces for the same problem, preference learning effectively identifies the specific logical pivots that lead to correct outcomes.
  • Self-Supervised Parity: Training on self-generated consensus (Debate-MV) achieved performance comparable to training on ground-truth human labels, suggesting a scalable path for model improvement in domains where labels are scarce.

Results and Discussion

MACA led to substantial performance leaps across all tested configurations.

Accuracy gains across different models and benchmarks using MACA variants.

As shown in the results, the MV-DPO variant consistently outperformed other methods across different debate stages. On the MATH benchmark, collaborative performance improved by 26.87%. Crucially, these gains translated back to single-agent inference; for instance, a post-trained Llama-8B saw a 21.51% accuracy increase on MathQA without needing a debate at test time. Furthermore, the models became significantly more self-consistent, with a 27.6% improvement on GSM8K. The models also showed "strong legs," improving by +16.3% on GPQA and +11.6% on CommonsenseQA, despite never seeing those datasets during the MACA post-training.

Limitations and Future Work

The effectiveness of MACA is bounded by a "competence threshold"—the base model must be capable of reaching a correct consensus at least occasionally to generate a valid training signal. Additionally, there is a theoretical risk of reinforcing popular but incorrect biases if the majority consistently converges on an error. Future directions include exploring debates between different types of models (heterogeneous agents) and confidence-weighted voting to further refine the training signal.

Conclusion

Multi-Agent Consensus Alignment demonstrates that LLMs can effectively "teach themselves" by simulating a collaborative environment. By internalizing the results of these debates through preference alignment, models achieve higher accuracy, better consistency, and strong zero-shot generalization, paving the way for autonomous self-improving AI systems.

Summary updated

BibTeX
@inproceedings{samanta2025internalizing,
  author = {Ankur Samanta and Akshayaa Magesh and Youliang Yu and Runzhe Wu and Ayush Jain and Daniel Jiang and Boris Vidolov and Paul Sajda and Yonathan Efroni and Kaveh Hassani},
  title = {{Internalizing Self-Consistency in Language Models: Multi-Agent Consensus Alignment}},
  booktitle = {MALGAI @ ICLR},
  year = {2025},
  doi = {10.48550/arxiv.2509.15172},
}
Figure