Is Each Layer Non-trivial in CNN? (Student Abstract)

Wei Wang, Yanjie Zhu, Zhuoxu Cui, Dong LiangAAAI 2021 · DOI

Abstract

Deep neural networks like ResNet have achieved state-of-the-art performance but at the cost of massive parameter redundancy. This paper explores whether every layer in these networks is truly necessary. Through a "zero-out" ablation study on ResNet34 and PSPNet, the authors demonstrate that many layers are "trivial" — their removal has negligible impact on performance. Conversely, "feature decomposition layers" that change channel dimensions are identified as critical structural pillars. The study also reveals task-specific redundancy patterns: classification models exhibit high redundancy in deeper layers, whereas segmentation models rely heavily on them for pixel-level precision.

Introduction

The evolution of deep learning has favored increasingly deep and wide architectures. While models like ResNet perform exceptionally well, their high parameter counts raise questions about efficiency and necessity. Unlike traditional sequential networks (e.g., VGG) where every layer is a bottleneck, ResNets utilize skip connections that allow information to bypass specific layers. This research aims to distinguish "non-trivial" layers — those essential for performance — from redundant "trivial" layers that merely add computational overhead.

Methodology: The "Zero-Out" Test

To identify redundant components, the researchers employed a three-step experimental framework:

  1. Training: Standard ResNet34 (classification) and PSPNet-ResNet34 (segmentation) models were trained to baseline performance on Cifar-10 and T1 datasets.
  2. Ablation: Post-training, specific convolution kernels were replaced with zeros, effectively disabling those layers without further fine-tuning.
  3. Evaluation: The models were re-evaluated to measure the drop in accuracy (classification) or Dice score (segmentation).

The study targeted individual layers, continuous blocks, and specific "feature decomposition layers" responsible for changing feature map channel counts.

Key Findings

Task-Specific Sensitivity

The impact of layer removal varies significantly between classification and segmentation tasks. In classification (Cifar-10), the network relies heavily on early layers to learn global abstract features. Once these are established, many deeper layers become redundant.

Figure 1: Performance of Cifar-10 classification when individual layers are replaced by zero. The model maintains high accuracy when deeper layers are ablated.

In contrast, segmentation (T1) requires high-resolution spatial information, making the final layers of the network indispensable. Zeroing out the final block in the segmentation model caused the Dice score to drop to zero, whereas the classification model maintained over 80% accuracy even when its final blocks were disabled.

Figure 2: Performance of T1 segmentation when individual layers are replaced by zero. Green dots indicate non-trivial layers where performance drops significantly.

The Criticality of Feature Decomposition

The most "non-trivial" layers are those that change channel dimensions. These "feature decomposition layers" act as the structural backbone of the network. Replacing them with zeros consistently led to total model failure across both tasks. This suggests that while residual layers primarily perform feature adjustment or fine-tuning, the decomposition layers and identity mappings are responsible for the fundamental information flow and storage.

Conclusion and Future Work

This study provides empirical evidence that modern CNNs are highly over-parameterized, with many layers serving as "trivial" components. Identifying these layers could lead to more efficient training and pruning strategies. Future research should investigate if these patterns persist in non-residual architectures like Transformers and explore automated methods for identifying and removing trivial layers during the training phase to optimize model efficiency from the outset.

Summary updated

BibTeX
@inproceedings{wang2021is,
  author = {Wei Wang and Yanjie Zhu and Zhuoxu Cui and Dong Liang},
  title = {{Is Each Layer Non-trivial in CNN? (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17954},
}
Figure