Abstract
Deep neural networks like ResNet have achieved state-of-the-art performance but at the cost of massive parameter redundancy. This paper explores whether every layer in these networks is truly necessary. Through a "zero-out" ablation study on ResNet34 and PSPNet, the authors demonstrate that many layers are "trivial" — their removal has negligible impact on performance. Conversely, "feature decomposition layers" that change channel dimensions are identified as critical structural pillars. The study also reveals task-specific redundancy patterns: classification models exhibit high redundancy in deeper layers, whereas segmentation models rely heavily on them for pixel-level precision.
Introduction
The evolution of deep learning has favored increasingly deep and wide architectures. While models like ResNet perform exceptionally well, their high parameter counts raise questions about efficiency and necessity. Unlike traditional sequential networks (e.g., VGG) where every layer is a bottleneck, ResNets utilize skip connections that allow information to bypass specific layers. This research aims to distinguish "non-trivial" layers — those essential for performance — from redundant "trivial" layers that merely add computational overhead.
Methodology: The "Zero-Out" Test
To identify redundant components, the researchers employed a three-step experimental framework:
- Training: Standard ResNet34 (classification) and PSPNet-ResNet34 (segmentation) models were trained to baseline performance on Cifar-10 and T1 datasets.
- Ablation: Post-training, specific convolution kernels were replaced with zeros, effectively disabling those layers without further fine-tuning.
- Evaluation: The models were re-evaluated to measure the drop in accuracy (classification) or Dice score (segmentation).
The study targeted individual layers, continuous blocks, and specific "feature decomposition layers" responsible for changing feature map channel counts.
Key Findings
Task-Specific Sensitivity
The impact of layer removal varies significantly between classification and segmentation tasks. In classification (Cifar-10), the network relies heavily on early layers to learn global abstract features. Once these are established, many deeper layers become redundant.
In contrast, segmentation (T1) requires high-resolution spatial information, making the final layers of the network indispensable. Zeroing out the final block in the segmentation model caused the Dice score to drop to zero, whereas the classification model maintained over 80% accuracy even when its final blocks were disabled.
The Criticality of Feature Decomposition
The most "non-trivial" layers are those that change channel dimensions. These "feature decomposition layers" act as the structural backbone of the network. Replacing them with zeros consistently led to total model failure across both tasks. This suggests that while residual layers primarily perform feature adjustment or fine-tuning, the decomposition layers and identity mappings are responsible for the fundamental information flow and storage.
Conclusion and Future Work
This study provides empirical evidence that modern CNNs are highly over-parameterized, with many layers serving as "trivial" components. Identifying these layers could lead to more efficient training and pruning strategies. Future research should investigate if these patterns persist in non-residual architectures like Transformers and explore automated methods for identifying and removing trivial layers during the training phase to optimize model efficiency from the outset.
Summary updated