Abstract
Knowledge distillation from LLMs is essential for the efficient deployment of language models. Prior works have proposed data generation using LLMs for preparing distilled models. We argue that generating data with LLMs is prone to sampling mainly from the center of original content distribution. This limitation hinders the distilled model from learning the true underlying data distribution and to forget the tails of the distributions (samples with lower probability). To this end, we propose GOLD, a task-agnostic data generation and knowledge distillation framework, which employs an iterative out-of-distribution-guided feedback mechanism for the LLM. As a result, the generated data improves the generalizability of distilled models. An energy-based OOD evaluation approach is also introduced to deal with noisy generated data. Our extensive experiments on 10 different classification and sequence-to-sequence tasks in NLP show that GOLD respectively outperforms prior arts and the LLM with an average improvement of 5% and 14%. We will also show that the proposed method is applicable to less explored and novel tasks.
Introduction
While Large Language Models (LLMs) like LLaMA or GPT-4 exhibit remarkable capabilities, their massive scale presents significant deployment challenges, including high computational costs and privacy risks associated with external API calls. Knowledge Distillation (KD) offers a solution by transferring the intelligence of these "teacher" models into smaller, more efficient "student" models (SLMs).
However, traditional data-free distillation methods suffer from a critical flaw: LLMs tend to generate "safe," high-probability data that clusters around the center of the training distribution. By ignoring the low-probability "tail" events—complex or rare scenarios—the resulting SLMs become brittle and fail to generalize. To address this, we introduce GOLD (Generalized Knowledge Distillation), a framework designed to force the teacher LLM to generate data that specifically targets the student's weaknesses through an iterative feedback loop.
The GOLD Framework
GOLD treats data generation as a competitive, iterative process between the teacher (LLM) and the student (SLM). The framework consists of three primary components: training data generation, validation data generation, and out-of-distribution (OOD) sample selection.
Iterative Feedback Loop
Instead of a static dataset, GOLD operates in rounds. In each iteration, the LLM is prompted to generate a validation set that is intentionally distinct in style and topic from the existing training data. The SLM then processes this validation set, and its performance is evaluated using an Energy-Based Model (EBM) approach.
By calculating the "free energy score" from the SLM's output logits, the system identifies samples that the student finds confusing (high energy). These "failure modes" are fed back into the LLM's prompt for the next round, steering it to generate more data in those difficult regions of the distribution. To handle potential noise in synthetic labels, GOLD utilizes Symmetric Cross-Entropy (SCE) loss, ensuring stable training even with imperfect teacher guidance.
Experimental Results
We evaluated GOLD across a diverse array of NLP benchmarks, including natural language inference (ANLI, MNLI, RTE), reading comprehension (SQuAD), and specialized tasks like linear programming formulation (NL4OPT).
Performance Gains
GOLD consistently outperformed state-of-the-art data-free distillation methods such as ZeroGen and ProGen. On average, GOLD-distilled models achieved a 5% improvement across classification tasks. Notably, the distilled SLM (T5-base) often surpassed the few-shot performance of its own teacher (LLaMA2-7B) by an average of 14%, suggesting that the iterative diversity of GOLD provides a more comprehensive learning signal than static few-shot prompting.
Diversity and Robustness
Ablation studies confirmed that the OOD feedback loop is essential for performance. Without it, the LLM tends to repeat common patterns that the student already understands. By targeting the "tails" of the distribution, GOLD ensures the student model develops a more robust and generalized understanding of the task. This was particularly evident in complex tasks like SVAMP (math word problems) and NL4OPT, where GOLD achieved high ROUGE-L and Exact Match scores compared to baselines.
Conclusion
GOLD demonstrates that the key to effective knowledge distillation lies not just in the volume of data, but in its diversity and relevance to the student's current knowledge gaps. By using energy-based OOD detection to guide the teacher LLM, GOLD creates a more robust student model capable of handling complex, real-world distributions. Future work will explore automated data valuation to further refine the feedback loop and extend the framework to multimodal domains.
Summary updated