MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models

Justin Chih-Yao Chen, Swarnadeep Saha, Elias Stengel-Eskin, Mohit Bansal

Introduction

Recent advancements in Large Language Models (LLMs) have demonstrated that multi-agent frameworks—where multiple models engage in collaborative reasoning or debate—can significantly outperform individual models. Systems like ReConcile leverage the "wisdom of the crowd" by allowing models such as GPT-4, Bard, and Claude to refine their answers through multi-round discussions. However, these frameworks are often impractical for production environments due to their extreme computational costs, high latency, and massive token requirements.

To bridge the gap between the high performance of multi-agent systems and the efficiency of smaller, standalone models, we present MAGDi (Structured Distillation of Multi-Agent Interaction Graphs). MAGDi is a novel distillation method designed to capture the complex dynamics of multi-agent interactions—including corrections, disagreements, and structural refinements—and compress them into a single, compact student model.

Methodology

The core innovation of MAGDi lies in its representation of the reasoning process as a Multi-Agent Interaction Graph (MAG). Unlike traditional distillation, which focuses on a single teacher's final output, MAGDi treats the entire debate as a structured learning signal.

1. Graph Representation

In a MAG, each node represents a specific reasoning chain (Chain-of-Thought) generated by an agent during a specific round of discussion. Edges represent the flow of information, such as one agent responding to or building upon the previous thoughts of another. This structure preserves the context of how a consensus was reached or how errors were corrected.

2. GCN-Augmented Training

During the training phase, the student model (e.g., Mistral-7B) is augmented with a Graph Convolutional Network (GCN). This GCN is tasked with learning "structure-aware" representations of the reasoning chains based on their specific positions and roles within the debate graph. This allows the student to understand not just the text, but the process of logical refinement.

3. Optimization Objectives

MAGDi employs a three-pronged loss function to train the student model:

  • Next-Token Prediction: The standard objective of learning from correct reasoning paths.
  • Contrastive Loss: A "negative reasoning" objective where the model learns to distinguish between correct paths and the incorrect paths generated during the multi-agent debate. This helps the model identify and avoid common logical pitfalls.
  • Graph-Based Classification: A structural objective that forces the model to internalize the interaction patterns within the MAG.

4. Zero-Shot Inference

Crucially, the GCN and graph-based objectives are only utilized during training. At inference time, the student model operates as a standard, standalone LLM. This ensures that the model retains its original architecture and efficiency while benefiting from the distilled multi-agent wisdom.

Experimental Setup

The researchers evaluated MAGDi across seven diverse benchmarks spanning commonsense and mathematical reasoning:

  • Commonsense Reasoning: StrategyQA, CommonsenseQA, ARC-Challenge, and BoolQ (Out-of-Domain).
  • Math Reasoning: GSM8K, MATH, and SVAMP (Out-of-Domain).

The training set consisted of 1,000 MAGs per in-domain task, generated using a multi-agent ensemble of GPT-4, Bard, and Claude 2.

Key Results and Insights

Performance and Efficiency

MAGDi achieved a significant performance leap, improving the base Mistral-7B model's accuracy by an average of 10.71% across all benchmarks. Notably, MAGDi outperformed the strongest single-teacher distillation baseline (distilling solely from GPT-4) by 4.61%.

In terms of efficiency, MAGDi provides a 9x reduction in token count compared to the multi-agent system it was distilled from. It effectively delivers the accuracy of a complex ensemble within a single, fast inference call.

Robustness and Generalization

The benefits of MAGDi extend beyond the specific tasks seen during training. In out-of-domain evaluations, such as the SVAMP math dataset, the multi-task version (MAGDi-MT) outperformed single-teacher baselines by 7%. This suggests that the model is learning generalized reasoning patterns rather than just memorizing task-specific answers.

The Value of Negative Signals

A key finding of the study is that "negative reasoning" is a powerful signal. By observing how a teacher model failed and was subsequently corrected by its peers, the student model learns more robust boundaries for logical consistency. Traditional distillation methods that discard incorrect answers miss this critical context.

Discussion and Limitations

While MAGDi represents a significant step forward, it is not without limitations. The initial generation of Multi-Agent Interaction Graphs is computationally expensive, requiring multiple calls to high-end proprietary APIs. Additionally, there is a risk that student models may inherit biases or hallucination patterns present in the teacher discussions.

Future work may explore more dynamic or fluid graph structures, allowing the student to learn from less rigid interaction patterns. Furthermore, investigating how the student's performance scales as the diversity and size of the teacher "crowd" increases remains an open area of research.

Conclusion

MAGDi demonstrates that the complex interactions of multi-agent systems can be effectively distilled into smaller, efficient models. By treating reasoning as a structured graph and leveraging both positive and negative signals, MAGDi allows 7B-parameter models to punch significantly above their weight class, offering a path toward high-performance reasoning in resource-constrained environments.

Summary updated

BibTeX
@inproceedings{chen2024magdi,
  author = {Justin Chih-Yao Chen and Swarnadeep Saha and Elias Stengel-Eskin and Mohit Bansal},
  title = {{MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=ffLblkoCw8},
}
Figure