Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining

Yasutoshi Ida, Yasuhiro Fujiwara

Abstract

Residual Networks with convolutional layers are widely used in the field of machine learning. Since they effectively extract features from input data by stacking multiple layers, they can achieve high accuracy in many applications. However, the stacking of many layers raises their computation costs. To address this problem, we propose Network Implosion, it erases multiple layers from Residual Networks without degrading accuracy. Our key idea is to introduce a priority term that identifies the importance of a layer; we can select unimportant layers according to the priority and erase them after the training. In addition, we retrain the networks to avoid critical drops in accuracy after layer erasure. A theoretical assessment reveals that our erasure and retraining scheme can erase layers without accuracy drop, and achieve higher accuracy than is possible with training from scratch. Our experiments show that Network Implosion can, for classification on Cifar-10/100 and ImageNet, reduce the number of layers by 24.00 to 42.86 percent without any drop in accuracy.

Introduction

Deep Residual Networks (ResNets) have become the industry standard for computer vision due to their high accuracy, but their "deeper is better" philosophy comes at a high computational cost. While we can train these massive models on clusters of GPUs, running them (inference) on resource-constrained devices like smartphones or IoT sensors remains a significant challenge.

Existing solutions often fall into two camps:

  1. Dynamic Pruning: Skipping layers during runtime. This saves computation but doesn't reduce the model's memory footprint because the skipped layers still occupy space.
  2. Static Pruning: Removing weights or filters. While effective, completely removing entire layers often leads to a "cliff-edge" drop in accuracy that is difficult to recover.

To address these limitations, we propose Network Implosion (NI)—a framework designed to permanently remove entire blocks of layers to reduce both memory and computation while maintaining, or even exceeding, the original model's performance.

Methodology: The Erasure and Retraining Cycle

Network Implosion centers on a systematic cycle of identifying unimportant layers, erasing them, and retraining the remaining structure to adapt to the change.

Identifying Layer Importance

To decide which layers to cut, we introduce a learnable scalar weight ($w_l$) for each Residual Unit. During initial training, the network learns which units contribute most to the final prediction. Units with a weight near zero are deemed "unimportant" and become candidates for erasure.

Iterative Erasure

Instead of pruning the network in a single pass, NI gradually erases the $k$ least important layers. This iterative approach prevents the structural shock that typically causes accuracy to crash in static pruning. We found that the first Residual Unit of any "stage" (where image resolution drops and channels increase) is critical for generating new feature representations and should never be pruned.

High-Learning Rate Retraining

When a layer is removed, the network's internal structure shifts drastically. Using a large learning rate during the retraining phase is essential to "shake" the remaining parameters into their new optimal configuration. This allows the remaining layers to compensate for the lost representational capacity of the erased units.

Theoretical Foundation

Our theoretical assessment utilizes the theory of hyperplane arrangements to analyze the network's representational power. While pruning reduces the total number of regions the network can define in the input space, the generalization error bound (via Rademacher average) actually tightens. Essentially, a smaller, well-retrained network can generalize better to new data than a massive, over-parameterized one, provided the remaining parameters are optimized correctly.

Experimental Results

We evaluated Network Implosion on three standard computer vision benchmarks: Cifar-10, Cifar-100, and ImageNet (ILSVRC2012). The results demonstrate significant efficiency gains across the board.

Efficiency and Accuracy

  • Layer Reduction: The method reduced the number of layers by 24.00% to 42.86% without any drop in accuracy.
  • ImageNet Performance: On the large-scale ImageNet dataset, the compressed model (38 layers) achieved 76.12% accuracy, actually outperforming the original 50-layer baseline (75.89%).
  • Computational Gains: The "imploded" ImageNet model reduced the number of Multiply-Accumulate operations (MACs) by approximately 21% and improved forward propagation speed by roughly 23%.

Comparison with Knowledge Distillation

NI consistently outperformed traditional "Teacher-Student" training (Knowledge Distillation), especially as models became shallower. While Knowledge Distillation accuracy tended to crash when the student model became too small, NI maintained high performance by leveraging the existing weights of the larger model.

Discussion: Preferable Initial Parameters

A key insight from our research is the Preferable Initial Parameters hypothesis. An imploded network often outperforms a network of the same size trained from scratch. We argue that starting with a deep, pre-trained model provides a set of initial parameters that guide the smaller network toward a superior local minimum that random initialization cannot reach.

Conclusion and Future Work

Network Implosion provides a robust pathway for compressing ResNets into leaner, faster models without sacrificing accuracy. While the iterative retraining cycle increases total training time, the resulting efficiency gains during inference are substantial. Future work will explore the application of this "implosion" logic to non-residual architectures, such as DenseNets and Vision Transformers (ViTs), and investigate automated methods for tuning the erasure hyperparameter $k$.

Summary updated

Abstract

Residual Networks with convolutional layers are widely used in the field of machine learning. Since they effectively extract features from input data by stacking multiple layers, they can achieve high accuracy in many applications. However, the stacking of many layers raises their computation costs. To address this problem, we propose Network Implosion, it erases multiple layers from Residual Networks without degrading accuracy. Our key idea is to introduce a priority term that identifies the importance of a layer; we can select unimportant layers according to the priority and erase them after the training. In addition, we retrain the networks to avoid critical drops in accuracy after layer erasure. A theoretical assessment reveals that our erasure and retraining scheme can erase layers without accuracy drop, and achieve higher accuracy than is possible with training from scratch. Our experiments show that Network Implosion can, for classification on Cifar-10/100 and ImageNet, reduce the number of layers by 24.00 to 42.86 percent without any drop in accuracy.
BibTeX
@inproceedings{ida2019network,
  author = {Yasutoshi Ida and Yasuhiro Fujiwara},
  title = {{Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining}},
  booktitle = {IJCNN},
  year = {2019},
  doi = {10.1109/ijcnn.2019.8852442},
}
Figure