How to Trade Off the Quantity and Capacity of Teacher Ensemble: Learning Categorical Distribution to Stochastically Employ a Teacher for Distillation

Zixiang Ding, Guoqing Jiang, Shuai Zhang, Lin Guo, Wei LinAAAI 2024 · DOI

TL;DR

This paper introduces DynaKD, a novel knowledge distillation method that learns an adaptive categorical distribution to dynamically select teachers from an ensemble. This approach effectively balances teacher quantity and capacity, achieving state-of-the-art performance in BERT compression across various NLP tasks and demonstrating generalization to image classification.

What Problem Does This Paper Address?

The paper tackles a fundamental challenge in knowledge distillation (KD) for large language models like BERT: how to best leverage an ensemble of teachers to train a smaller student model. While using more teachers or stronger teachers intuitively seems beneficial, empirical evidence suggests otherwise. Specifically, simply averaging or combining multiple teachers can lead to a loss of individual teacher diversity, and using a single, very powerful teacher can result in a "capacity gap" where the student struggles to learn from its overly complex mentor. Existing methods like SKDBERT attempt to address this by stochastically sampling teachers, but require manually designing the sampling distribution, which is time-consuming and suboptimal. This paper aims to automate and optimize this teacher selection process.

Methods

The core contribution is Dynamic Knowledge Distillation (DynaKD). DynaKD introduces an adaptive categorical distribution that stochastically selects a teacher from an ensemble at each distillation step. This distribution is learned through a bi-level optimization framework:

  1. Categorical Distribution Search (Upper Level): This phase optimizes the categorical distribution parameters ($\theta$) to minimize the student's performance on a validation set. It uses a differentiable approach, employing continuous relaxation and a single-step adaptation scheme (borrowed from meta-learning and NAS) to approximate the inner optimization.
  2. Student Distillation (Lower Level): In this phase, the learned categorical distribution is fixed, and the student model's parameters ($w$) are trained to minimize the distillation loss, using the sampled teachers according to the probabilities defined by the categorical distribution.

This bi-level optimization allows DynaKD to dynamically determine the importance of each teacher for different downstream tasks, effectively balancing the "quantity" (using an ensemble) and "capacity" (selecting the right teacher for the job) of the teacher ensemble.

Datasets

The paper evaluates DynaKD on the following datasets:

  • GLUE Benchmark: This includes MRPC, RTE, CoLA, SST-2, QQP, QNLI, and MNLI. The standard GLUE development set (GLUE-dev) is used for evaluating the learned categorical distributions.
  • CIFAR-100: Used to demonstrate the generalization capabilities of DynaKD to computer vision tasks.

Key Insights and Ideas

  • Teacher Ensemble Optimization is Crucial: The paper strongly reinforces that simply having more or stronger teachers isn't a silver bullet for knowledge distillation. The way teachers are selected and utilized is paramount.
  • Adaptive Teacher Selection Outperforms Static Methods: DynaKD's learned adaptive categorical distribution significantly outperforms hand-crafted distributions (uniform, teacher-rank, student-rank) across all tested tasks, highlighting the benefit of task-specific teacher importance.
  • Bridging the Capacity Gap: DynaKD effectively uses "assistant teachers" (those with intermediate capacities) to help transfer knowledge from the strongest teacher, mitigating the capacity gap issue.
  • Differentiable Optimization for Teacher Selection: The bi-level optimization framework provides a principled and efficient way to learn the optimal teacher selection strategy, making the process automatic and data-driven.
  • Generalization Beyond NLP: Demonstrating state-of-the-art performance on CIFAR-100 shows that DynaKD's principles are applicable beyond BERT-style models and NLP tasks.

Key Results

  • State-of-the-Art on GLUE: DynaKD achieves state-of-the-art performance on six out of seven GLUE benchmark tasks.
  • On MRPC, it achieved 92.2 F1 score and 89.1 accuracy, improving by 1.4 and 2.1 points respectively over the previous best (MT-BERT).
  • On RTE, it achieved a 4.0 point absolute improvement.
  • Competitive on CIFAR-100: DynaKD also achieved state-of-the-art performance for image classification on CIFAR-100.
  • Computational Efficiency: DynaKD's categorical distribution search and evaluation process is significantly more efficient than TinyBERT's multi-stage distillation, with a total cost of 22.42 hours compared to TinyBERT's 249.56 hours on five downstream tasks.

Limitations and Future Work

  • Computational Cost of Search: While more efficient than TinyBERT, the categorical distribution search phase still incurs a computational cost. The paper mentions that DynaKD delivers candidates every m steps, suggesting potential for further optimization in the search process.
  • Generalization to Diverse Teacher Ensembles: The current work focuses on BERT-style teachers. Future work could explore DynaKD's effectiveness with more diverse teacher architectures or even different model families.
  • Exploring Different Optimization Schemes: The paper uses a single-step adaptation for the inner optimization. Investigating more sophisticated optimization techniques for the bi-level problem could potentially yield further improvements.
  • Hyperparameter Sensitivity: While the paper provides experimental settings, the sensitivity of DynaKD to various hyperparameters (e.g., learning rates, batch sizes for search vs. distillation) could be a point for future investigation.

Summary updated

BibTeX
@inproceedings{ding2024how,
  author = {Zixiang Ding and Guoqing Jiang and Shuai Zhang and Lin Guo and Wei Lin},
  title = {{How to Trade Off the Quantity and Capacity of Teacher Ensemble: Learning Categorical Distribution to Stochastically Employ a Teacher for Distillation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29746},
}
Figure