GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation

Mohsen Gholami, Mohammad Akbari, Cindy Hu, Vaden Masrani, Z. Jane Wang, Yong Zhang

Abstract

GOLD (Generalized Knowledge Distillation) is a novel framework designed to improve the distillation of knowledge from massive Large Language Models (LLMs) into smaller, more efficient Small Language Models (SLMs). By employing an iterative feedback loop guided by "out-of-distribution" (OOD) samples, GOLD forces the teacher LLM to generate diverse, challenging data that covers the "tails" of the distribution. This approach significantly boosts the student model's performance and generalizability across a wide range of NLP tasks.

Problem Statement

While LLMs like LLaMA or GPT-4 are highly capable, their immense size makes them expensive and slow to deploy in production environments. Knowledge Distillation (KD) aims to solve this by training a "Student" SLM to mimic a "Teacher" LLM.

A popular approach is "data-free" distillation, where the LLM generates its own synthetic training data. However, LLMs tend to be "boring" when generating data, naturally sampling from the high-probability "center" of a distribution while ignoring rare but critical "tail" events. This results in student models that perform well on common cases but fail on unusual or complex inputs.

Methodology

The authors introduce GOLD, which stands for Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation. The framework treats data generation as a dynamic, iterative process rather than a one-off event.

Detailed architecture of the GOLD framework showing the iterative loop between training data generation, student model evaluation, and OOD-based feedback.

The process consists of three main stages per iteration:

  1. Train Data Generation: The LLM ($\mathcal{M}_L$) generates a batch of training data ($X_{train}$) based on a task definition and a few real-world examples.
  2. SLM Training & Evaluation: The student model ($\mathcal{M}_S$) is trained on this data. Simultaneously, the LLM generates a separate "OOD validation set" ($X_{val}$) designed to be stylistically and topically distinct from the training set.
  3. OOD-Based Feedback: GOLD uses an energy-based evaluation approach to identify samples in the validation set where the student model is most "confused" (high-energy/OOD samples). These failure modes are fed back into the LLM's prompt for the next round, steering it to generate more challenging samples.

To handle the inevitable noise (incorrect labels) in synthetic data, the authors utilize a Symmetric Cross-Entropy (SCE) loss function, which provides greater robustness than standard training objectives.

Datasets and Evaluation

GOLD was evaluated across 10 diverse NLP tasks:

  • Classification: ANLI, MNLI, QNLI, WNLI, RTE, and MRPC.
  • Sequence-to-Sequence: SQuAD and Adv-QA (Question Answering), and SVAMP (Math word problems).
  • Specialized Domains: NL4OPT, a dataset for formulating linear programming optimization problems from text descriptions.

Key Insights

  • Capturing the Long Tail: Vanilla LLM generation often creates a "collapsed" distribution. Analysis of token distributions reveals that GOLD successfully pushes generation toward a long-tailed distribution that more closely matches real-world data diversity.
  • Unsupervised Failure Detection: By using energy scores to identify OOD samples, the system detects student weaknesses without requiring ground-truth labels for synthetic validation data, making the feedback loop self-contained.
  • Task Agnosticism: Unlike previous methods hard-coded for classification, GOLD’s prompt-based feedback works for complex sequence-to-sequence tasks like math and optimization.

Results

GOLD consistently outperformed prior state-of-the-art distillation methods (such as ZeroGen and ProGen) and, in many cases, exceeded the few-shot performance of the teacher LLM itself.

Performance comparison of GOLD against various baselines on standard NLU classification benchmarks.
  • Classification: GOLD achieved an average accuracy of 67.1% across six tasks, a 4% improvement over prior arts and a nearly 15% jump over the teacher LLM's few-shot performance.
  • Seq-to-Seq: On the SQuAD dataset, GOLD reached 75.2% Exact Match (EM), outperforming the next best baseline (P2Model) by nearly 1%.
  • Novel Tasks: On the NL4OPT task, GOLD achieved a ROUGE-L score of 72.8, proving its adaptability to specialized domains where the teacher LLM might not have been explicitly fine-tuned.

Conclusion

GOLD provides a robust framework for distilling LLM knowledge into efficient small models. By closing the loop between data generation and model evaluation through OOD-guided feedback, it enables the creation of SLMs that generalize better to complex, real-world scenarios. Future work will focus on automated data filtering and extending this framework to multimodal tasks.

Summary updated

BibTeX
@inproceedings{gholami2024gold,
  author = {Mohsen Gholami and Mohammad Akbari and Cindy Hu and Vaden Masrani and Z. Jane Wang and Yong Zhang},
  title = {{GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=ULm8WqgkVI},
}
Figure