Abstract
Knowledge Distillation (KD) is a standard technique for compressing computationally expensive pre-trained language models (PLMs) into smaller, more efficient versions. However, traditional KD typically results in a "performance gap," where the smaller student model fails to match the original teacher's capabilities. We introduce Co-Training and Co-Distillation (CTCD), a framework that facilitates bidirectional knowledge transfer between models during training. By allowing the teacher to learn from the student while simultaneously instructing it, CTCD breaks the performance ceiling of traditional distillation. Our results demonstrate that a 4-layer student model distilled via CTCD can outperform its original 6-layer teacher by 1.66 points on the GLUE benchmark, effectively achieving both compression and quality improvement.
Introduction
The deployment of large-scale language models like BERT or GPT in resource-constrained environments—such as mobile devices or real-time applications—is often hindered by their massive computational requirements. While Knowledge Distillation has emerged as a popular solution, it is traditionally a one-way process: a fixed, pre-trained "teacher" provides targets for a "student." This static relationship limits the student's potential to the teacher's pre-existing knowledge.
This paper proposes a paradigm shift: What if the teacher and student learned together? By treating distillation as a collaborative process, we can improve the performance of both models. The CTCD framework addresses the inherent limitations of one-way KD by fostering a feedback loop that enhances the teacher's accuracy, which in turn provides higher-quality guidance to the student.
The CTCD Framework
The core of the CTCD framework is the simultaneous training of two or more models with mutual knowledge exchange. Unlike standard KD, which uses a frozen teacher, CTCD involves training both models from the pre-training phase.
Bidirectional Distillation
CTCD employs two primary distillation paths:
- Standard KD: The student model learns to match the probability distributions (soft targets) of the teacher.
- Reversed KD: The teacher model is updated based on the student's outputs. This allows the teacher to adapt its internal representations based on the student's learning progress.
Loss Function Synergy
Each model in the CTCD framework optimizes a composite loss function consisting of:
- Hard Loss: A standard cross-entropy loss based on the ground truth labels.
- Soft Loss: A distillation loss that encourages the model to match the output distribution of its counterpart.
Community KD
For more complex scenarios, the authors introduce Community KD. In this setup, a pre-trained teacher guides multiple students who also engage in co-distillation with one another. This "classroom" environment allows students to benefit from diverse perspectives, further boosting the robustness of the compressed models.
Experimental Setup
To evaluate the effectiveness of CTCD, the researchers utilized a 30-million sentence subset of BookCorpus and Wikipedia for the pre-training phase. The models were then evaluated on the GLUE (General Language Understanding Evaluation) benchmark, which covers nine diverse natural language processing tasks, including sentiment analysis (SST-2), linguistic acceptability (CoLA), and natural language inference (MNLI).
Key Results and Discussion
The experimental results challenge the conventional wisdom that compression must come at the cost of quality.
Outperforming the Teacher
The most significant finding was that a 4-layer BERT student model trained via CTCD achieved a GLUE score of 79.12. This outperformed the original 6-layer standalone teacher model (78.06) by 1.66 points. In contrast, traditional one-way distillation at the same training scale only provided a marginal gain of 0.33 points.
The Virtuous Cycle
The success of CTCD stems from a positive feedback loop. As the teacher model "explains" concepts to the student and receives feedback via Reversed KD, its own performance improves—averaging an increase of 1.88 points on GLUE. This improved teacher then becomes a more effective instructor, pushing the student beyond the performance ceiling of the original teacher architecture.
Efficiency Gains
While CTCD requires more initial computational resources than one-way KD (approximately 3 GPU days compared to 2.3 days), the resulting models offer a superior trade-off between inference speed and accuracy. The framework is also orthogonal to other techniques like data augmentation or specialized architecture design, meaning it can be integrated into existing pipelines to further enhance performance.
Conclusion
Co-Training and Co-Distillation (CTCD) represents a significant advancement in model compression. By transforming distillation into a bidirectional learning process, CTCD enables the creation of smaller models that are not only faster but also more accurate than their larger predecessors. Future work will explore "Architecture Sharing" to reduce the memory footprint during the co-training phase and the integration of CTCD with task-specific data augmentation strategies.
Summary updated