Abstract
Multi-exit architectures improve inference efficiency by allowing "easy" samples to exit early, reducing computational costs. However, training these models is challenging due to objective interference between exits and the difficulty of manual loss weight tuning. This paper introduces Harmonized Dense Knowledge Distillation (HDKD), a framework where every exit learns from all subsequent exits simultaneously. By employing a bilevel optimization strategy, HDKD automatically tunes the importance of each loss term, achieving superior performance across various datasets and computational budgets.
Introduction
Deep neural networks are often computationally expensive, limiting their deployment on resource-constrained devices. Multi-exit architectures, such as MSDNet, address this by placing auxiliary classifiers at different depths. While promising, training these models faces three primary hurdles:
- Objective Interference: Early exits require simple features, while later exits need complex, abstract representations, leading to conflicting gradients.
- Suboptimal Supervision: Traditional methods often use only the final exit as a teacher, ignoring the "dark knowledge" available in intermediate layers.
- Manual Tuning: Determining the optimal weights for multiple exit losses is a tedious process that rarely yields the best possible performance.
Methods
HDKD addresses these challenges through a two-pronged approach: Dense Knowledge Distillation and Harmonized Weighting.
Dense Knowledge Distillation
Instead of relying solely on the final classifier, HDKD implements a dense supervision scheme. For any exit $i$, every subsequent exit $j$ (where $j > i$) acts as a teacher. This ensures that early layers receive a rich gradient signal derived from multiple levels of abstraction throughout the network.
Harmonized Weighting via Bilevel Optimization
To eliminate manual weight tuning, the authors treat loss weights ($\gamma$) as learnable hyperparameters. They utilize a bilevel optimization strategy:
- Inner Loop: Network parameters ($\theta$) are updated on the training set using the current weights.
- Outer Loop: Loss weights ($\gamma$) are optimized based on performance on a separate validation set.
To prevent the optimizer from over-focusing on early exits (which naturally have higher error rates), a ratio regularizer is introduced to balance the relative improvement across all exits.
Key Results
The researchers evaluated HDKD using the MSDNet architecture on CIFAR100 and ImageNet (ILSVRC 2012), including various low-data subsets.
Accuracy and Efficiency
HDKD consistently outperformed state-of-the-art baselines. On ImageNet(100), it improved the first exit's accuracy by 8.56% over the "Impr" baseline. In budgeted batch classification scenarios, HDKD provided higher accuracy for the same computational cost (FLOPs) compared to MSDNet and other distillation variants.
Robustness and Insights
- Intermediate Teachers: Results suggest that intermediate exits often provide more "digestible" hints for shallow layers than the final exit alone.
- Dynamic Weighting: Bilevel optimization allows the model to dynamically adjust which teachers are most influential at different stages of training.
- Initialization Independence: The method proved robust even with randomized initial loss weights, as the optimization process successfully converged to effective values.
Conclusion
Harmonized Dense Knowledge Distillation (HDKD) provides a robust framework for training multi-exit networks by leveraging dense supervision and automated weight tuning. While the bilevel optimization introduces additional training overhead due to second-order gradient calculations, the resulting models offer significant improvements in inference efficiency and accuracy. Future work may explore further resolving conflicts between early and late exits in more diverse multi-task learning scenarios.
Summary updated