VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks

Susan Gao, Xin Liu, Lung-Sheng Chien, William Zhang, José M. ÁlvarezWorkshops @ ICCV 2019 · DOI · arXiv

Introduction

Deep Residual Networks (ResNets) have become the backbone of modern computer vision. However, their significant computational requirements make deployment on edge devices challenging. While model pruning—the removal of redundant weights—is a standard solution, ResNets pose unique difficulties. Specifically, the element-wise addition in skip-connections requires that pruned channels be perfectly aligned across layers to achieve actual hardware speedups. Furthermore, standard group-sparsity methods often fail when weight magnitudes within a group vary significantly, leading to "incomplete" pruning where a few large weights prevent a group from being fully zeroed out.

The VACL Methodology

To address these challenges, the authors introduce Variance-Aware Cross-Layer Regularization (VACL). This approach moves beyond layer-wise pruning by considering the global structure of the residual blocks.

Cross-Layer Grouping

VACL identifies all layers coupled by skip-connections and groups corresponding filters into single regularization units. As shown in the architectural diagram below, the $i$-th filters across multiple layers are treated as a single weight group ($W_i$). This ensures that if a specific channel is deemed redundant, it is removed across the entire block, maintaining the structural integrity required for efficient hardware execution.

Cross-layer weight grouping in VACL ensures structural alignment across residual blocks.

Variance-Aware Regularization

The core innovation of VACL is its "Variance-Aware" penalty. Traditional Group Lasso pushes groups toward zero but ignores the internal distribution of weight magnitudes. VACL introduces a term that minimizes the variance of weight magnitudes (absolute values) within each group. By forcing weights to have similar scales, the optimizer can more effectively push an entire group to zero simultaneously, preventing a few "stubborn" large weights from keeping a low-importance group alive.

Key Insights: Magnitude Similarity and Optimization

A critical finding of this research is that minimizing the variance of weight magnitudes is superior to minimizing the variance of raw weight values. This choice fundamentally alters the optimization landscape and creates a more stable pruning process.

Comparison of feasible sets for different regularization terms; (d) shows the VACL 'four-sharp-corner' set that encourages magnitude similarity.

As illustrated in the figure above, different regularization terms result in different "feasible sets" in the weight space. While standard L1 (a) and L2 (b) norms create diamonds or circles, the VACL penalty (d) creates a "four-sharp-corner" set. This geometry effectively encourages structural sparsity by pushing weights toward a state where they are either all zero or share similar magnitudes, leading to a more reliable "all-or-nothing" pruning effect.

Experimental Results

VACL was evaluated on CIFAR-10, CIFAR-100, and ImageNet benchmarks, demonstrating state-of-the-art performance:

  • ResNet-110 (CIFAR-10): Achieved a 79.5% reduction in model size with no loss in accuracy (and in some cases, a slight improvement).
  • ResNeXt-29 (CIFAR-10): Reached an 82% pruning ratio with less than 1% accuracy degradation.
  • ResNet-50 (ImageNet): Successfully pruned 63.3% of the model while maintaining top-5 accuracy within 1% of the baseline.

Beyond raw compression, VACL-pruned models showed superior transferability. When a ResNet-50 pruned on ImageNet was used for scene recognition on the Indoor-67 dataset, it outperformed other popular pruning methods like ThiNet. This suggests that VACL preserves more generalizable and meaningful features during the compression process.

Conclusion and Future Directions

VACL provides a robust framework for pruning ResNets by aligning filters across skip-connections and minimizing intra-group magnitude variance. While highly effective, the method currently requires manual tuning of the regularization strength ($\lambda$) and is optimized specifically for element-wise addition topologies. Future work may explore automated hyperparameter optimization and extending the variance-aware principle to more complex architectures like DenseNets or those discovered via Neural Architecture Search (NAS).

Summary updated

BibTeX
@inproceedings{gao2019vacl,
  author = {Susan Gao and Xin Liu and Lung-Sheng Chien and William Zhang and José M. Álvarez},
  title = {{VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  doi = {10.1109/iccvw.2019.00360},
}
Figure