Staged Knowledge Distillation Through Least-to-Most Prompting: Optimizing Teacher Guidance via Difficulty-Aware Training

Mengxiang Zhang, Lingyuan LiuEMNLP 2025

TL;DR

This paper introduces Least-to-Most Prompting Knowledge Distillation (L2M-KD), a novel method for compressing large language models (LLMs) by intelligently guiding a smaller "student" model with a larger "teacher" model. L2M-KD uses a curriculum learning approach to present training examples from easy to hard and dynamically adjusts the distillation loss, achieving superior performance without relying on computationally expensive student-generated outputs.

What Problem Does This Paper Address?

Large Language Models (LLMs) are incredibly powerful but also computationally expensive, limiting their deployment in resource-constrained environments. Knowledge Distillation (KD) is a technique to create smaller, more efficient models by transferring knowledge from a large teacher LLM to a smaller student LLM. However, existing "white-box" KD methods, which have access to the teacher's internal workings, face challenges:

  • Training-Inference Mismatches: Using only ground-truth data can lead to discrepancies between how the model is trained and how it performs in practice.
  • Suboptimal Performance: Standard KD loss functions like Kullback-Leibler Divergence (KLD) can lead to "mode-averaging" and "over-smoothing," where the student model's outputs become too generalized and lose important nuances.
  • Computational Overhead: Some advanced KD techniques rely on student-generated outputs, which can be computationally expensive to generate and use.

The authors aim to develop a more efficient and effective white-box KD method that overcomes these limitations, particularly for instruction-following tasks.

Methods

L2M-KD combines two key ideas: Curriculum Learning (CL) and Adaptive Loss Design.

  1. Least-to-Most Prompting Curriculum Learning:
    • Instead of training on the entire dataset at once, L2M-KD ranks training samples by difficulty. This difficulty is measured using Rouge-L scores, comparing student-generated outputs (from a preliminary student model) with ground-truth outputs.
  2. The dataset is then partitioned into multiple "easy-to-hard" subsets.
  3. Training proceeds in stages, starting with the easiest subset and gradually incorporating more difficult ones. This mimics how humans learn, building foundational knowledge before tackling complex problems.
  4. Adaptive KD Loss Function:
    • The distillation loss function dynamically transitions from Kullback-Leibler Divergence (KLD) to Skew-KLD (SKL).
  5. KLD is good for simpler tasks, while SKL, which introduces a smoothing mechanism, is better for more complex ones.
  6. By starting with KLD on easy samples and moving to SKL on harder samples, the method provides appropriate teacher guidance that adapts to the student's evolving learning stage. This helps mitigate mode-averaging and over-smoothing.
  7. The method also uses an adaptive ratio ($\beta_i$) to balance the KD loss and the standard cross-entropy loss, further optimizing the training process.

Crucially, L2M-KD exclusively uses ground-truth outputs (GTOs), avoiding the computational cost associated with student-generated outputs.

Datasets

  • Training: databricks-dolly-15k dataset, containing 15,000 human-written instruction-response pairs.
  • Evaluation: Five instruction-following datasets: DollyEval, SelfInst, VicunaEval, S-NI, and UnNI.

Key Insights and Ideas

  • Difficulty-Aware Training is Key: Ranking and progressively introducing training samples based on difficulty significantly improves knowledge transfer efficiency.
  • Dynamic Loss Functions Mimic Learning: The transition from KLD to SKL mirrors a natural learning process, providing appropriate guidance as the student model tackles more complex tasks.
  • GTOs are Sufficient: The method demonstrates that high-quality distillation can be achieved using only ground-truth outputs, bypassing the computational burden of student-generated outputs.
  • Least-to-Most Prompting Principle: Applying this educational strategy to KD leads to more stable and effective knowledge transfer.

Key Results

L2M-KD consistently outperformed existing white-box KD methods (SFT, SeqKD, KLD, SKL) across various GPT-2 and OPT teacher-student configurations.

  • Superior Performance: L2M-KD achieved higher average Rouge-L scores, often surpassing the teacher model itself on certain datasets, especially with larger student models. For example, with the OPT-1.3B student, L2M-KD achieved scores of 30.1 and 35.8 on S-NI and UnNI, respectively, compared to the teacher's 19.2 and 22.7.
  • Efficiency: By using only GTOs, L2M-KD avoids the computational overhead of on-policy methods.
  • Robustness: The method showed effectiveness across different model families (GPT-2, OPT) and was also validated on larger models like OpenLLaMA2 (7B teacher, 2B student).

Limitations and Future Work

  • Limited Model Families and Tasks: Experiments were primarily conducted on GPT-2 and OPT for instruction-following tasks. Future work should explore L2M-KD's generalizability to other architectures (e.g., Qwen 2.5, Llama 3.2) and diverse NLP tasks like text generation and summarization.
  • Student Model Size: The current study focused on smaller student models. For very large student models, the perceived difficulty of samples might be less varied, potentially impacting the CL strategy's effectiveness. Future work will investigate this with larger student models.

Summary updated

BibTeX
@inproceedings{zhang2025staged,
  author = {Mengxiang Zhang and Lingyuan Liu},
  title = {{Staged Knowledge Distillation Through Least-to-Most Prompting: Optimizing Teacher Guidance via Difficulty-Aware Training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure