TL;DR
This paper introduces a novel knowledge distillation technique using optimal transport (OT) to align feature distributions between a large "teacher" neural network and a smaller "student" network. By minimizing the OT cost between intermediate layer features, the student network learns to mimic the teacher's feature representations more effectively, leading to improved accuracy without significant increases in computational cost.
What Problem Does This Paper Address?
Deploying powerful deep learning models, especially in computer vision, is often hindered by their substantial computational and memory requirements. This is particularly problematic for resource-constrained devices like mobile phones. Model compression techniques aim to reduce model size and complexity while preserving accuracy. Knowledge distillation (KD) is a popular method where a smaller "student" network learns from a larger, pre-trained "teacher" network. While standard KD focuses on matching the final output probabilities, this paper addresses the limitation that it doesn't fully leverage the rich intermediate feature representations learned by the teacher. Existing methods that do leverage intermediate features, like FitNets or Relational Knowledge Distillation (RKD), have their own drawbacks, such as being too rigid or too loose in their constraints. The paper seeks to find a more balanced and effective way to transfer knowledge from intermediate layers.
Methods
The core contribution is the development of optimal transport-based loss functions for knowledge distillation. Specifically, they propose using the Wasserstein-1 distance (also known as Earth Mover's Distance) to measure the "distance" between the feature distributions of the teacher and student networks at various intermediate layers.
- Optimal Transport (OT) Loss: The proposed loss function, $L_{OT}(X^{(l)}, Y^{(l)})$, aims to minimize the cost of transforming the student's feature distribution ($Y^{(l)}$) into the teacher's feature distribution ($X^{(l)}$) at layer $l$. This is achieved by minimizing a cost matrix $C$ (where $C_{ij}$ is the distance between teacher feature $x_i$ and student feature $y_j$) weighted by a transport matrix $T$. The cosine distance on the unit sphere is used as the cost function.
- Combined Loss: This OT loss is then combined with the standard cross-entropy loss ($L_{CE}$) for training the student network: $L = L_{CE} + \alpha \sum L_{OT}$. Additional losses like KD and Contrastive Representation Distillation (CRD) can also be incorporated.
- Efficient OT Computation: To address the computational expense of exact OT, the paper explores two approximations:
- Inexact Proximal Optimal Transport (IPOT): An efficient algorithm based on inexact proximal point iterations, offering a good balance between training time and accuracy.
- Relaxed Earth-Mover's Distance (REMD): A computationally cheaper relaxation of the OT problem.
Datasets
The paper evaluates its methods on the following standard computer vision datasets:
- CIFAR-100: A dataset of 50,000 training images and 10,000 test images, with 100 fine-grained categories.
- ImageNet (ILSVRC2012): A large-scale dataset with 1.2 million training images and 50,000 validation images, categorized into 1000 classes. The validation set is used as the test set.
- SVHN (Street View House Numbers): A dataset of 73,257 training images and 26,032 test images, containing cropped images of house numbers with 10 digit classes.
Key Insights and Ideas
- Feature Distribution Alignment is Key: The paper's central insight is that directly matching the distributions of intermediate features, rather than just the final outputs or pairwise geometric relationships, provides a more effective way to transfer knowledge.
- Optimal Transport as a Flexible Metric: OT offers a principled way to compare distributions that may not have overlapping support, making it suitable for comparing feature representations that can vary significantly between teacher and student networks.
- Soft Alignment Advantage: Unlike rigid methods that force exact feature map matching, OT allows for a "soft alignment," where student features can be influenced by multiple teacher features, offering greater flexibility and potentially better generalization.
- Layer-wise Contribution: The study shows that even using OT loss on a single intermediate layer can yield significant improvements, though combining multiple layers generally leads to better results.
- Complementary to Other Methods: The proposed OT loss is presented as complementary to existing KD techniques like KD and CRD, and combining them often yields superior performance.
Key Results
- Improved Accuracy: The proposed IPOT and REMD losses, especially when combined with KD (IPOT+KD, REMD+KD), consistently outperform or match other state-of-the-art knowledge distillation methods on CIFAR-100, ImageNet, and SVHN datasets.
- Competitive Performance on ImageNet: On ImageNet, IPOT+KD achieves performance comparable to CRD+KD, which is noted as the state-of-the-art.
- SVHN Success: IPOT+KD achieves state-of-the-art performance on SVHN for one teacher-student pair and is the second-best for another.
- Efficiency Trade-offs: While IPOT is computationally more demanding than REMD, both offer practical solutions for incorporating OT into the training pipeline. The paper notes that IPOT training can take around 10 hours on a GTX 1080.
Limitations and Future Work
- Computational Cost: While IPOT and REMD are presented as efficient solutions, the exact OT computation is still computationally prohibitive for large batch sizes. The paper acknowledges that MMD is more scalable for large batches.
- Hyperparameter Tuning: The performance of OT-based losses can be sensitive to hyperparameters like $\alpha$ and $\gamma$, which balance the different loss terms.
- Future Directions:
- Contrastive OT: The authors suggest exploring "contrastive" optimal transport loss functions, inspired by CRD, to further enhance knowledge distillation by pushing apart negative pairs and pulling together positive pairs.
- Faster OT Methods: Investigating and employing even faster methods for comparing distributions, such as interpolated distances between MMD and OT, is proposed.
- Optimal Combination of Losses: Finding the optimal combination and weighting of various loss terms (OT, KD, CRD) for different scenarios is an area for future research.
Summary updated