Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining

Yasutoshi Ida, Yasuhiro Fujiwara

TL;DR

"Network Implosion" is a new compression technique that statically prunes entire layers from Residual Networks (ResNets) and retrains the remaining structure. By identifying unimportant layers through a learned priority term, the method reduces layer counts by up to 42% on ImageNet without losing accuracy—and in some cases, actually improving it.

What Problem Does This Paper Address?

Deep Residual Networks (ResNets) have become the industry standard for computer vision due to their high accuracy, but their "deeper is better" philosophy comes at a high computational cost. While we can train these massive models on clusters of GPUs, running them (inference) on resource-constrained devices like smartphones or IoT sensors is difficult.

Existing solutions often fall into two camps:

  1. Dynamic Pruning: Skipping layers during runtime. This saves computation but doesn't reduce the model's memory footprint because the skipped layers still occupy space.
  2. Static Pruning: Removing weights or filters. While effective, completely removing entire layers often leads to a "cliff-edge" drop in accuracy that is hard to recover.

The authors seek a way to "implode" the network—permanently removing entire blocks of layers to reduce both memory and computation while maintaining (or exceeding) the original model's performance.

Methods

The authors propose a framework called Network Implosion (NI), which centers on a cycle of "erasure and retraining."

  • Identifying Importance: To decide which layers to cut, they introduce a learnable scalar weight ($w_l$) for each Residual Unit. During initial training, the network learns which units are contributing most to the final prediction. Units with a weight near zero are deemed "unimportant."
  • The "Erasure and Retraining" Cycle: Instead of pruning everything at once, NI gradually erases the $k$ least important layers and then retrains the remaining network using a high learning rate. This allows the remaining layers to adapt to the structural change.
  • Theoretical Backing: The authors use the theory of hyperplane arrangements to show that while pruning reduces the total "representational power," the generalization error bound (via Rademacher average) actually tightens. Essentially, a smaller, well-retrained network can generalize better to new data than a massive, over-parameterized one.

Datasets

The researchers evaluated Network Implosion on three standard computer vision benchmarks:

  • Cifar-10: 10 classes of small $32\times32$ images.
  • Cifar-100: 100 classes of small images.
  • ImageNet (ILSVRC2012): The gold standard for large-scale image classification, featuring 1,000 classes and high-resolution images.

Key Insights and Ideas

  • The "Preferable Initial Parameters" Hypothesis: One of the most surprising findings is that the imploded network often outperforms a network of the same size trained from scratch. The authors argue that starting with a deep, pre-trained model provides "preferable initial parameters" that guide the smaller network to a better solution than random initialization ever could.
  • Don't Touch the "Stage" Transitions: ResNets are organized into stages where the image resolution drops and the number of channels increases. The authors found that the first Residual Unit of any stage is critical for accuracy and should never be pruned, as it is responsible for generating new feature representations.
  • Retraining with High Learning Rates: When a layer is removed, the network's internal structure shifts drastically. Using a large learning rate during the retraining phase is essential to "shake" the remaining parameters into their new optimal configuration.

Key Results

Network Implosion demonstrated significant efficiency gains across all tested datasets:

  • Layer Reduction: The method reduced the number of layers by 24.00% to 42.86% without any drop in accuracy.
  • Accuracy Boost: On ImageNet, the compressed model (38 layers) actually achieved 76.12% accuracy, outperforming the original 50-layer baseline (75.89%).
  • Computational Efficiency: On ImageNet, the "imploded" model reduced the number of Multiply-Accumulate operations (MACs) by roughly 21% and sped up forward propagation time by approximately 23%.
  • Superiority over Knowledge Distillation: NI consistently outperformed "Teacher-Student" training (Knowledge Distillation), especially as the models became shallower. While Knowledge Distillation accuracy crashed when models got too small, NI maintained high performance.

Limitations and Future Work

  • Training Time: Because NI relies on an iterative "prune and retrain" cycle, the total training time is significantly longer than training a standard model once.
  • Architecture Specificity: The current study focuses specifically on ResNets. While the logic of residual connections makes this pruning possible, it remains to be seen how well this "implosion" logic applies to non-residual architectures like DenseNets or Vision Transformers (ViTs).
  • Hyperparameter Sensitivity: The choice of $k$ (how many layers to erase at once) and the retraining epoch count are manual settings that might require tuning for different tasks.

Summary updated

Abstract

Residual Networks with convolutional layers are widely used in the field of machine learning. Since they effectively extract features from input data by stacking multiple layers, they can achieve high accuracy in many applications. However, the stacking of many layers raises their computation costs. To address this problem, we propose Network Implosion, it erases multiple layers from Residual Networks without degrading accuracy. Our key idea is to introduce a priority term that identifies the importance of a layer; we can select unimportant layers according to the priority and erase them after the training. In addition, we retrain the networks to avoid critical drops in accuracy after layer erasure. A theoretical assessment reveals that our erasure and retraining scheme can erase layers without accuracy drop, and achieve higher accuracy than is possible with training from scratch. Our experiments show that Network Implosion can, for classification on Cifar-10/100 and ImageNet, reduce the number of layers by 24.00 to 42.86 percent without any drop in accuracy.
BibTeX
@inproceedings{ida2019network,
  author = {Yasutoshi Ida and Yasuhiro Fujiwara},
  title = {{Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining}},
  booktitle = {IJCNN},
  year = {2019},
  doi = {10.1109/ijcnn.2019.8852442},
}

Loading related content...

Figure