Improving Reasoning Capabilities in Small Models through Mixture-of-layers Distillation with Stepwise Attention on Key Information

Yao Chen, Jiawei Sheng, Wenyuan Zhang, Tingwen Liu

Abstract

The significant computational demands of large language models have increased interest in distilling reasoning abilities into smaller models via Chain-of-Thought (CoT) distillation. Current CoT distillation methods mainly focus on transferring teacher-generated rationales for complex reasoning to student models. However, they do not adequately explore teachers’ dynamic attention toward critical information during reasoning. We find that language models exhibit progressive attention shifts towards key information during reasoning, which implies essential clues for drawing conclusions. Building on this observation and analysis, we introduce a novel CoT distillation framework that transfers the teacher’s stepwise attention on key information to the student model. This establishes structured guidance for the student’s progressive concentration on key information during reasoning. More importantly, we develop a Mixture of Layers module enabling dynamic alignment that adapts to different layers between the teacher and student. Our method achieves consistent performance improvements across multiple mathematical and commonsense reasoning datasets. To our knowledge, it is the first method to leverage stepwise attention within CoT distillation to improve small model reasoning.

Introduction

Large Language Models (LLMs) like Llama-3 have demonstrated remarkable capabilities in Chain-of-Thought (CoT) reasoning, enabling them to solve complex problems by breaking them down into intermediate steps. However, the deployment of these models is often restricted by their massive computational requirements. While CoT distillation aims to transfer these reasoning abilities to smaller "student" models, traditional approaches primarily focus on mimicking the teacher's textual output. This often leaves the student model without an understanding of the underlying "focus" or attention patterns the teacher utilized to reach a conclusion. Consequently, small models frequently struggle with multi-step reasoning where tracking specific critical information, such as numerical values in math problems, is essential.

The MoLSAKI Framework

To address these limitations, we propose MoLSAKI (Mixture-of-Layers Distillation with Stepwise Attention on Key Information). This framework moves beyond simple rationale transfer by teaching the student model how to pay attention to critical tokens during each step of the reasoning process.

The MoLSAKI framework architecture, illustrating CoT distillation, stepwise attention extraction, and the Mixture of Layers module for adaptive alignment.

The framework consists of two primary components:

  1. Stepwise Attention on Critical Tokens (SAKI): We observed that teacher models exhibit "progressive attention shifts," where their focus moves between different critical tokens (e.g., numbers) as they progress through reasoning steps. MoLSAKI extracts these attention maps and uses KL Divergence to guide the student model in mimicking these focus patterns.
  2. Mixture-of-Layers (MoL) Alignment: Since teacher and student models often have different numbers of layers, a rigid one-to-one mapping is ineffective. MoL employs a dynamic "router" to aggregate information across layers. It prioritizes the teacher's intermediate layers—where the most significant reasoning occurs—and learns a weighted combination of student layers to match this focus.

Analysis of Stepwise Attention

Our analysis confirms that during multi-step reasoning, models do not attend to all information equally at all times. Instead, there is a clear diagonal trend in attention, where specific steps correlate strongly with specific critical tokens in the input.

Heatmap showing the stepwise attention of a model on critical tokens (N1-N12) across different reasoning steps (S1-S5), demonstrating progressive focus shifts.

This structured guidance provides the student with a "roadmap" for reasoning, ensuring that it concentrates on the relevant data points at the correct time, rather than losing track of information in complex sequences.

Experimental Results

We evaluated MoLSAKI across several mathematical (SVAMP, GSM8K, AsDiv) and commonsense (CSQA) reasoning benchmarks. The results demonstrate consistent improvements:

  • Performance Gains: TinyLlama (1.1B) showed an 11.3% average relative improvement over standard distillation baselines. GPT2-Large (774M) improved by 7.5%.
  • Generalization: The framework exhibited strong out-of-domain performance. For instance, models trained on SVAMP maintained their lead when tested on the more challenging GSM8K dataset, suggesting the acquisition of general reasoning skills rather than simple pattern memorization.
  • Efficiency: The authors note that the MoL module adds negligible computational overhead during training compared to the performance gains it provides.

Conclusion

MoLSAKI introduces a novel dimension to CoT distillation by leveraging the internal attention mechanisms of teacher models. By teaching small models to mimic the stepwise focus of larger counterparts, we significantly enhance their reasoning accuracy and robustness. Future work will explore scaling this approach to more complex, long-form reasoning tasks and diverse model architectures.

Summary updated

Abstract

The significant computational demands of large language models have increased interest in distilling reasoning abilities into smaller models via Chain-of-Thought (CoT) distillation. Current CoT distillation methods mainly focus on transferring teacher-generated rationales for complex reasoning to student models. However, they do not adequately explore teachers’ dynamic attention toward critical information during reasoning. We find that language models exhibit progressive attention shifts towards key information during reasoning, which implies essential clues for drawing conclusions. Building on this observation and analysis, we introduce a novel CoT distillation framework that transfers the teacher’s stepwise attention on key information to the student model. This establishes structured guidance for the student’s progressive concentration on key information during reasoning. More importantly, we develop a Mixture of Layers module enabling dynamic alignment that adapts to different layers between the teacher and student. Our method achieves consistent performance improvements across multiple mathematical and commonsense reasoning datasets. To our knowledge, it is the first method to leverage stepwise attention within CoT distillation to improve small model reasoning.
BibTeX
@inproceedings{chen2025improving,
  author = {Yao Chen and Jiawei Sheng and Wenyuan Zhang and Tingwen Liu},
  title = {{Improving Reasoning Capabilities in Small Models through Mixture-of-layers Distillation with Stepwise Attention on Key Information}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=mFHaagYywm},
}
Figure