Weighted Mutual Learning with Diversity-Driven Model Compression

Miao Zhang, Li Wang, David Gonzalo Chaves Campos, Wei Huang, Chenjuan Guo, Bin YangNeurIPS 2022 · OpenReview

TL;DR

This paper introduces Weighted Mutual Learning with Diversity-Driven Model Compression (WML), a novel framework for online knowledge distillation. WML tackles the challenges of memory consumption and lack of diversity in existing online distillation methods by using structured network pruning to create varied student models and a bi-level optimization approach to dynamically weigh their contributions.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are powerful but computationally expensive, limiting their deployment on edge devices. Knowledge distillation (KD) aims to transfer knowledge from large "teacher" models to smaller "student" models. While traditional KD (offline) requires a pre-trained teacher, online distillation trains multiple student models collaboratively, allowing them to learn from each other. However, online distillation faces two main hurdles:

  1. Memory Consumption: Training multiple large models simultaneously requires significant memory.
  2. Lack of Diversity: If student models are too similar, they may not learn sufficiently diverse knowledge from each other, hindering overall performance.

This paper proposes WML to address these issues, aiming to improve the efficiency and effectiveness of online knowledge distillation.

Methods

WML introduces two key innovations:

  1. Diversity-Driven Model Compression: Instead of using identical student models, WML employs structured network pruning (specifically, SNIP-based filter pruning) to create a set of "peer" models with varying architectures and sizes. This not only introduces diversity but also significantly reduces memory requirements by allowing peers to share parts of their architecture in a hierarchical manner.
  2. Weighted Mutual Learning with Bi-level Formulation: Traditional online distillation often averages the knowledge from peers. WML, however, treats the learning process as a bi-level optimization problem. The "inner loop" trains the peer models using a weighted loss function, while the "outer loop" dynamically learns the optimal weights for each peer. This weighted approach allows the framework to adaptively determine the importance of each peer, prioritizing those that contribute more effectively to the overall learning process. The paper derives a closed-form solution for the hypergradient, enabling efficient weight updates.

Datasets

The paper evaluates WML on the following datasets:

  • CIFAR10
  • CIFAR100
  • ImageNet

Key Insights and Ideas

  • Diversity is Key: Introducing architectural diversity among student models, even through pruning, significantly enhances the effectiveness of online knowledge distillation.
  • Dynamic Weighting Outperforms Averaging: Simply averaging peer knowledge is suboptimal. Dynamically learning peer importance allows the distillation process to leverage the most valuable insights from each student.
  • Pruning as a Feature: Structured network pruning, typically used for compression, can be repurposed as a method to generate diverse student models for improved distillation.
  • Efficiency Gains: WML achieves competitive results while also producing a set of compressed models as a byproduct, offering a multi-faceted solution.

Key Results

WML demonstrates strong performance across various networks and datasets:

  • CIFAR100: WML achieved significant accuracy boosts over self-distillation baselines (e.g., 2.29% to 3.56% improvement on ResNet18, ResNet50, and ResNet101).
  • ImageNet: WML outperformed self-distillation baselines on ImageNet as well.
  • Comparison with Online Distillation: WML generally outperformed other online distillation methods (RKD, CTSL-MKT, DML, ONE, SD) on CIFAR10, CIFAR100, and ImageNet.
  • Channel Pruning Comparison: As a byproduct, the pruned models generated by WML achieved competitive or superior accuracy compared to existing channel pruning methods, often with fewer FLOPs. For instance, on CIFAR10, WML models showed minimal accuracy drops (e.g., -0.36% for WML30 on ResNet32) while achieving significant FLOP reduction (24.4%).

Limitations and Future Work

  • Pruning Ratio Sensitivity: While diversity is beneficial, excessively high pruning ratios can degrade peer model performance and negatively impact distillation. The paper suggests finding an optimal pruning range.
  • Training-Based Pruning: The current framework uses a training-free pruning method (SNIP). The authors suggest that incorporating more advanced training-based channel pruning methods could further boost performance and efficiency.
  • Future Directions: The authors propose extending WML to feature distillation, applying it to other domains like object detection and natural language processing, and exploring architecture search. They also suggest investigating the impact of more sophisticated pruning techniques and a wider range of peer model diversities.

Summary updated

Abstract

Online distillation attracts attention from the community as it simplifies the traditional two-stage knowledge distillation process into a single stage. Online distillation collaboratively trains a group of peer models, which are treated as students, and all students gain extra knowledge from each other. However, memory consumption and diversity among peers are two key challenges to the scalability and quality of online distillation. To address the two challenges, this paper presents a framework called Weighted Mutual Learning with Diversity-Driven Model Compression (WML) for online distillation. First, at the base of a hierarchical structure where peers share different parts, we leverage the structured network pruning to generate diversified peer models and reduce the memory requirements. Second, rather than taking the average of peers, this paper, for the first time, leverages a bi-level formulation to estimate the relative importance of peers with a close-form, to further boost the effectiveness of the distillation from each other. Extensive experiments show the generalization of the proposed framework, which outperforms existing online distillation methods on a variety of deep neural networks. More interesting, as a byproduct, \WML produces a series of pruned models under different model sizes in a single run, which also achieves competitive results compared with existing channel pruning methods.
BibTeX
@inproceedings{zhang2022weighted,
  author = {Miao Zhang and Li Wang and David Gonzalo Chaves Campos and Wei Huang and Chenjuan Guo and Bin Yang},
  title = {{Weighted Mutual Learning with Diversity-Driven Model Compression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=UQJoGBNRX4},
}
Figure