Introduction
Deep Residual Networks (ResNets) have become the backbone of modern computer vision. However, their significant computational requirements make deployment on edge devices challenging. While model pruning—the removal of redundant weights—is a standard solution, ResNets pose unique difficulties. Specifically, the element-wise addition in skip-connections requires that pruned channels be perfectly aligned across layers to achieve actual hardware speedups. Furthermore, standard group-sparsity methods often fail when weight magnitudes within a group vary significantly, leading to "incomplete" pruning where a few large weights prevent a group from being fully zeroed out.
The VACL Methodology
To address these challenges, the authors introduce Variance-Aware Cross-Layer Regularization (VACL). This approach moves beyond layer-wise pruning by considering the global structure of the residual blocks.
Cross-Layer Grouping
VACL identifies all layers coupled by skip-connections and groups corresponding filters into single regularization units. As shown in the architectural diagram below, the $i$-th filters across multiple layers are treated as a single weight group ($W_i$). This ensures that if a specific channel is deemed redundant, it is removed across the entire block, maintaining the structural integrity required for efficient hardware execution.
Variance-Aware Regularization
The core innovation of VACL is its "Variance-Aware" penalty. Traditional Group Lasso pushes groups toward zero but ignores the internal distribution of weight magnitudes. VACL introduces a term that minimizes the variance of weight magnitudes (absolute values) within each group. By forcing weights to have similar scales, the optimizer can more effectively push an entire group to zero simultaneously, preventing a few "stubborn" large weights from keeping a low-importance group alive.
Key Insights: Magnitude Similarity and Optimization
A critical finding of this research is that minimizing the variance of weight magnitudes is superior to minimizing the variance of raw weight values. This choice fundamentally alters the optimization landscape and creates a more stable pruning process.
As illustrated in the figure above, different regularization terms result in different "feasible sets" in the weight space. While standard L1 (a) and L2 (b) norms create diamonds or circles, the VACL penalty (d) creates a "four-sharp-corner" set. This geometry effectively encourages structural sparsity by pushing weights toward a state where they are either all zero or share similar magnitudes, leading to a more reliable "all-or-nothing" pruning effect.
Experimental Results
VACL was evaluated on CIFAR-10, CIFAR-100, and ImageNet benchmarks, demonstrating state-of-the-art performance:
- ResNet-110 (CIFAR-10): Achieved a 79.5% reduction in model size with no loss in accuracy (and in some cases, a slight improvement).
- ResNeXt-29 (CIFAR-10): Reached an 82% pruning ratio with less than 1% accuracy degradation.
- ResNet-50 (ImageNet): Successfully pruned 63.3% of the model while maintaining top-5 accuracy within 1% of the baseline.
Beyond raw compression, VACL-pruned models showed superior transferability. When a ResNet-50 pruned on ImageNet was used for scene recognition on the Indoor-67 dataset, it outperformed other popular pruning methods like ThiNet. This suggests that VACL preserves more generalizable and meaningful features during the compression process.
Conclusion and Future Directions
VACL provides a robust framework for pruning ResNets by aligning filters across skip-connections and minimizing intra-group magnitude variance. While highly effective, the method currently requires manual tuning of the regularization strength ($\lambda$) and is optimized specifically for element-wise addition topologies. Future work may explore automated hyperparameter optimization and extending the variance-aware principle to more complex architectures like DenseNets or those discovered via Neural Architecture Search (NAS).
Summary updated