GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation

Mohsen Gholami, Mohammad Akbari, Tianxi Hu, Vaden Masrani, Z. Wang, Yong Zhang

Abstract

Knowledge distillation from LLMs is essential for the efficient deployment of language models. Prior works have proposed data generation using LLMs for preparing distilled models. We argue that generating data with LLMs is prone to sampling mainly from the center of original content distribution. This limitation hinders the distilled model from learning the true underlying data distribution and to forget the tails of the distributions (samples with lower probability). To this end, we propose GOLD, a task-agnostic data generation and knowledge distillation framework, which employs an iterative out-of-distribution-guided feedback mechanism for the LLM. As a result, the generated data improves the generalizability of distilled models. An energy-based OOD evaluation approach is also introduced to deal with noisy generated data. Our extensive experiments on 10 different classification and sequence-to-sequence tasks in NLP show that GOLD respectively outperforms prior arts and the LLM with an average improvement of 5% and 14%. We will also show that the proposed method is applicable to less explored and novel tasks.

Introduction

While Large Language Models (LLMs) like LLaMA or GPT-4 exhibit remarkable capabilities, their massive scale presents significant deployment challenges, including high computational costs and privacy risks associated with external API calls. Knowledge Distillation (KD) offers a solution by transferring the intelligence of these "teacher" models into smaller, more efficient "student" models (SLMs).

However, traditional data-free distillation methods suffer from a critical flaw: LLMs tend to generate "safe," high-probability data that clusters around the center of the training distribution. By ignoring the low-probability "tail" events—complex or rare scenarios—the resulting SLMs become brittle and fail to generalize. To address this, we introduce GOLD (Generalized Knowledge Distillation), a framework designed to force the teacher LLM to generate data that specifically targets the student's weaknesses through an iterative feedback loop.

The GOLD Framework

GOLD treats data generation as a competitive, iterative process between the teacher (LLM) and the student (SLM). The framework consists of three primary components: training data generation, validation data generation, and out-of-distribution (OOD) sample selection.

The GOLD framework architecture, illustrating the iterative loop between LLM data generation and SLM feedback via OOD sample selection.

Iterative Feedback Loop

Instead of a static dataset, GOLD operates in rounds. In each iteration, the LLM is prompted to generate a validation set that is intentionally distinct in style and topic from the existing training data. The SLM then processes this validation set, and its performance is evaluated using an Energy-Based Model (EBM) approach.

By calculating the "free energy score" from the SLM's output logits, the system identifies samples that the student finds confusing (high energy). These "failure modes" are fed back into the LLM's prompt for the next round, steering it to generate more data in those difficult regions of the distribution. To handle potential noise in synthetic labels, GOLD utilizes Symmetric Cross-Entropy (SCE) loss, ensuring stable training even with imperfect teacher guidance.

Experimental Results

We evaluated GOLD across a diverse array of NLP benchmarks, including natural language inference (ANLI, MNLI, RTE), reading comprehension (SQuAD), and specialized tasks like linear programming formulation (NL4OPT).

Performance Gains

GOLD consistently outperformed state-of-the-art data-free distillation methods such as ZeroGen and ProGen. On average, GOLD-distilled models achieved a 5% improvement across classification tasks. Notably, the distilled SLM (T5-base) often surpassed the few-shot performance of its own teacher (LLaMA2-7B) by an average of 14%, suggesting that the iterative diversity of GOLD provides a more comprehensive learning signal than static few-shot prompting.

Performance comparison of GOLD against baseline distillation methods and the teacher LLM across various NLI benchmarks.

Diversity and Robustness

Ablation studies confirmed that the OOD feedback loop is essential for performance. Without it, the LLM tends to repeat common patterns that the student already understands. By targeting the "tails" of the distribution, GOLD ensures the student model develops a more robust and generalized understanding of the task. This was particularly evident in complex tasks like SVAMP (math word problems) and NL4OPT, where GOLD achieved high ROUGE-L and Exact Match scores compared to baselines.

Conclusion

GOLD demonstrates that the key to effective knowledge distillation lies not just in the volume of data, but in its diversity and relevance to the student's current knowledge gaps. By using energy-based OOD detection to guide the teacher LLM, GOLD creates a more robust student model capable of handling complex, real-world distributions. Future work will explore automated data valuation to further refine the feedback loop and extend the framework to multimodal domains.

Summary updated

Abstract

Knowledge distillation from LLMs is essential for the efficient deployment of language models. Prior works have proposed data generation using LLMs for preparing distilled models. We argue that generating data with LLMs is prone to sampling mainly from the center of original content distribution. This limitation hinders the distilled model from learning the true underlying data distribution and to forget the tails of the distributions (samples with lower probability). To this end, we propose GOLD, a task-agnostic data generation and knowledge distillation framework, which employs an iterative out-of-distribution-guided feedback mechanism for the LLM. As a result, the generated data improves the generalizability of distilled models. An energy-based OOD evaluation approach is also introduced to deal with noisy generated data. Our extensive experiments on 10 different classification and sequence-to-sequence tasks in NLP show that GOLD respectively outperforms prior arts and the LLM with an average improvement of 5% and 14%. We will also show that the proposed method is applicable to less explored and novel tasks. Code is available in the Appendix.
BibTeX
@inproceedings{gholami2024gold,
  author = {Mohsen Gholami and Mohammad Akbari and Tianxi Hu and Vaden Masrani and Z. Wang and Yong Zhang},
  title = {{GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=ULm8WqgkVI},
}
Figure