TL;DR
This paper introduces Weighted Mutual Learning with Diversity-Driven Model Compression (WML), a novel framework for online knowledge distillation. WML tackles the challenges of memory consumption and lack of diversity in existing online distillation methods by using structured network pruning to create varied student models and a bi-level optimization approach to dynamically weigh their contributions.
What Problem Does This Paper Address?
Deep neural networks (DNNs) are powerful but computationally expensive, limiting their deployment on edge devices. Knowledge distillation (KD) aims to transfer knowledge from large "teacher" models to smaller "student" models. While traditional KD (offline) requires a pre-trained teacher, online distillation trains multiple student models collaboratively, allowing them to learn from each other. However, online distillation faces two main hurdles:
- Memory Consumption: Training multiple large models simultaneously requires significant memory.
- Lack of Diversity: If student models are too similar, they may not learn sufficiently diverse knowledge from each other, hindering overall performance.
This paper proposes WML to address these issues, aiming to improve the efficiency and effectiveness of online knowledge distillation.
Methods
WML introduces two key innovations:
- Diversity-Driven Model Compression: Instead of using identical student models, WML employs structured network pruning (specifically, SNIP-based filter pruning) to create a set of "peer" models with varying architectures and sizes. This not only introduces diversity but also significantly reduces memory requirements by allowing peers to share parts of their architecture in a hierarchical manner.
- Weighted Mutual Learning with Bi-level Formulation: Traditional online distillation often averages the knowledge from peers. WML, however, treats the learning process as a bi-level optimization problem. The "inner loop" trains the peer models using a weighted loss function, while the "outer loop" dynamically learns the optimal weights for each peer. This weighted approach allows the framework to adaptively determine the importance of each peer, prioritizing those that contribute more effectively to the overall learning process. The paper derives a closed-form solution for the hypergradient, enabling efficient weight updates.
Datasets
The paper evaluates WML on the following datasets:
- CIFAR10
- CIFAR100
- ImageNet
Key Insights and Ideas
- Diversity is Key: Introducing architectural diversity among student models, even through pruning, significantly enhances the effectiveness of online knowledge distillation.
- Dynamic Weighting Outperforms Averaging: Simply averaging peer knowledge is suboptimal. Dynamically learning peer importance allows the distillation process to leverage the most valuable insights from each student.
- Pruning as a Feature: Structured network pruning, typically used for compression, can be repurposed as a method to generate diverse student models for improved distillation.
- Efficiency Gains: WML achieves competitive results while also producing a set of compressed models as a byproduct, offering a multi-faceted solution.
Key Results
WML demonstrates strong performance across various networks and datasets:
- CIFAR100: WML achieved significant accuracy boosts over self-distillation baselines (e.g., 2.29% to 3.56% improvement on ResNet18, ResNet50, and ResNet101).
- ImageNet: WML outperformed self-distillation baselines on ImageNet as well.
- Comparison with Online Distillation: WML generally outperformed other online distillation methods (RKD, CTSL-MKT, DML, ONE, SD) on CIFAR10, CIFAR100, and ImageNet.
- Channel Pruning Comparison: As a byproduct, the pruned models generated by WML achieved competitive or superior accuracy compared to existing channel pruning methods, often with fewer FLOPs. For instance, on CIFAR10, WML models showed minimal accuracy drops (e.g., -0.36% for WML30 on ResNet32) while achieving significant FLOP reduction (24.4%).
Limitations and Future Work
- Pruning Ratio Sensitivity: While diversity is beneficial, excessively high pruning ratios can degrade peer model performance and negatively impact distillation. The paper suggests finding an optimal pruning range.
- Training-Based Pruning: The current framework uses a training-free pruning method (SNIP). The authors suggest that incorporating more advanced training-based channel pruning methods could further boost performance and efficiency.
- Future Directions: The authors propose extending WML to feature distillation, applying it to other domains like object detection and natural language processing, and exploring architecture search. They also suggest investigating the impact of more sophisticated pruning techniques and a wider range of peer model diversities.
Summary updated