Co-training and Co-distillation for Quality Improvement and Compression of Language Models

Hayeon Lee, Rui Hou, Jongpil Kim, Davis Liang, Hongbo Zhang, Sung Ju Hwang, Alexander Min

Abstract

Knowledge Distillation (KD) is a standard technique for compressing computationally expensive pre-trained language models (PLMs) into smaller, more efficient versions. However, traditional KD typically results in a "performance gap," where the smaller student model fails to match the original teacher's capabilities. We introduce Co-Training and Co-Distillation (CTCD), a framework that facilitates bidirectional knowledge transfer between models during training. By allowing the teacher to learn from the student while simultaneously instructing it, CTCD breaks the performance ceiling of traditional distillation. Our results demonstrate that a 4-layer student model distilled via CTCD can outperform its original 6-layer teacher by 1.66 points on the GLUE benchmark, effectively achieving both compression and quality improvement.

Introduction

The deployment of large-scale language models like BERT or GPT in resource-constrained environments—such as mobile devices or real-time applications—is often hindered by their massive computational requirements. While Knowledge Distillation has emerged as a popular solution, it is traditionally a one-way process: a fixed, pre-trained "teacher" provides targets for a "student." This static relationship limits the student's potential to the teacher's pre-existing knowledge.

This paper proposes a paradigm shift: What if the teacher and student learned together? By treating distillation as a collaborative process, we can improve the performance of both models. The CTCD framework addresses the inherent limitations of one-way KD by fostering a feedback loop that enhances the teacher's accuracy, which in turn provides higher-quality guidance to the student.

The CTCD Framework

The core of the CTCD framework is the simultaneous training of two or more models with mutual knowledge exchange. Unlike standard KD, which uses a frozen teacher, CTCD involves training both models from the pre-training phase.

Bidirectional Distillation

CTCD employs two primary distillation paths:

  1. Standard KD: The student model learns to match the probability distributions (soft targets) of the teacher.
  2. Reversed KD: The teacher model is updated based on the student's outputs. This allows the teacher to adapt its internal representations based on the student's learning progress.

Loss Function Synergy

Each model in the CTCD framework optimizes a composite loss function consisting of:

  • Hard Loss: A standard cross-entropy loss based on the ground truth labels.
  • Soft Loss: A distillation loss that encourages the model to match the output distribution of its counterpart.

Community KD

For more complex scenarios, the authors introduce Community KD. In this setup, a pre-trained teacher guides multiple students who also engage in co-distillation with one another. This "classroom" environment allows students to benefit from diverse perspectives, further boosting the robustness of the compressed models.

Experimental Setup

To evaluate the effectiveness of CTCD, the researchers utilized a 30-million sentence subset of BookCorpus and Wikipedia for the pre-training phase. The models were then evaluated on the GLUE (General Language Understanding Evaluation) benchmark, which covers nine diverse natural language processing tasks, including sentiment analysis (SST-2), linguistic acceptability (CoLA), and natural language inference (MNLI).

Key Results and Discussion

The experimental results challenge the conventional wisdom that compression must come at the cost of quality.

Outperforming the Teacher

The most significant finding was that a 4-layer BERT student model trained via CTCD achieved a GLUE score of 79.12. This outperformed the original 6-layer standalone teacher model (78.06) by 1.66 points. In contrast, traditional one-way distillation at the same training scale only provided a marginal gain of 0.33 points.

The Virtuous Cycle

The success of CTCD stems from a positive feedback loop. As the teacher model "explains" concepts to the student and receives feedback via Reversed KD, its own performance improves—averaging an increase of 1.88 points on GLUE. This improved teacher then becomes a more effective instructor, pushing the student beyond the performance ceiling of the original teacher architecture.

Efficiency Gains

While CTCD requires more initial computational resources than one-way KD (approximately 3 GPU days compared to 2.3 days), the resulting models offer a superior trade-off between inference speed and accuracy. The framework is also orthogonal to other techniques like data augmentation or specialized architecture design, meaning it can be integrated into existing pipelines to further enhance performance.

Conclusion

Co-Training and Co-Distillation (CTCD) represents a significant advancement in model compression. By transforming distillation into a bidirectional learning process, CTCD enables the creation of smaller models that are not only faster but also more accurate than their larger predecessors. Future work will explore "Architecture Sharing" to reduce the memory footprint during the co-training phase and the integration of CTCD with task-specific data augmentation strategies.

Summary updated

Abstract

Knowledge Distillation (KD) compresses computationally expensive pre-trained language models (PLMs) by transferring their knowledge to smaller models, allowing their use in resource-constrained or real-time settings. However, most smaller models fail to surpass the performance of the original larger model, resulting in sacrificing performance to improve inference speed. To address this issue, we propose Co-Training and Co-Distillation (CTCD), a novel framework that improves performance and inference speed together by co-training two models while mutually distilling knowledge. The CTCD framework successfully achieves this based on two significant findings: 1) Distilling knowledge from the smaller model to the larger model during co-training improves the performance of the larger model. 2) The enhanced performance of the larger model further boosts the performance of the smaller model. The CTCD framework shows promise as it can be combined with existing techniques like architecture design or data augmentation, replacing one-way KD methods, to achieve further performance improvement. Extensive ablation studies demonstrate the effectiveness of CTCD, and the small model distilled by CTCD outperforms the original larger model by a significant margin of 1.66 on the GLUE benchmark.
BibTeX
@inproceedings{lee2023co,
  author = {Hayeon Lee and Rui Hou and Jongpil Kim and Davis Liang and Hongbo Zhang and Sung Ju Hwang and Alexander Min},
  title = {{Co-training and Co-distillation for Quality Improvement and Compression of Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=dJCAbMLobX},
}
Figure