Abstract
Convolutional neural network (CNN) models have achieved great success in many fields. With the advent of ResNet, networks used in practice are getting deeper and wider. However, is each layer non-trivial in networks? To answer this question, we trained a network on the training set, then we replaced the network convolution kernels with zeros and tested the resulting models on the test set. We compared experimental results with a baseline and showed that we can reach similar or even the same performances. Although convolution kernels are the cores of networks, we demonstrate that some of them are trivial and regular in ResNet.
Introduction
Since the introduction of ResNet, convolutional neural networks (CNNs) have grown increasingly deep and complex. While these architectures achieve state-of-the-art results, they often suffer from over-parameterization. We know these models are massive, but we don't always know if every layer is contributing meaningfully to the final output. This study investigates the distinction between "non-trivial" (essential) and "trivial" (redundant) layers, a distinction crucial for model compression, improving training efficiency, and gaining a deeper theoretical understanding of information flow through residual architectures.
Methodology
The researchers employed a straightforward "zeroing out" ablation strategy to identify layer importance:
- Training: Standard models (ResNet34 for classification and PSPNet-ResNet34 for segmentation) were trained to establish a baseline performance.
- Kernel Replacement: After training, specific convolution kernels ($w$) were replaced with zeros ($0$), and the modified model was evaluated on the test set.
- Ablation Groups: The study looked at single-layer ablation to see local impacts and block-wise ablation, which targeted sequences of layers with the same channel number.
By observing how performance metrics—Accuracy for classification and Dice score for segmentation—fluctuated, the researchers mapped out which parts of the architecture were most critical.
Task-Dependent Redundancy
The study evaluated two different types of computer vision tasks: image classification using the Cifar-10 dataset and image segmentation using a medical imaging dataset (T1).
In classification (Cifar-10), the later layers become increasingly trivial. Once the network has extracted global abstract features in the early-to-mid stages, the final layers do not appear to add significant value. As shown in the layer-wise analysis, the performance of the model with zeroed-out later layers eventually converges back toward the baseline accuracy of 84%.
In segmentation (T1), the back layers remain non-trivial. Because segmentation requires precise pixel-level information for the final output, the network cannot afford to "turn off" its later stages. The Dice score (baseline 87%) drops significantly when various layers across the entire depth of the network are zeroed out.
Key Insights: The "Feature Decomposition" Hypothesis
The most critical layers in a ResNet are those that change the channel dimensions, which the authors call "feature decomposition layers." When these specific layers are zeroed out, performance collapses regardless of the task. For instance, in Cifar-10, zeroing decomposition layers caused accuracy to plummet to as low as 16%, while in T1, the Dice score dropped to 0.00.
The widespread "triviality" of many other layers is likely a byproduct of ResNet’s architecture. Because ResNets use identity mappings (skip connections), the network can effectively "pass through" information if a residual layer isn't contributing, making the kernels in those layers redundant. This confirms that modern deep networks are significantly larger than they need to be for their assigned tasks.
Conclusion
This research demonstrates that many layers in deep ResNets are redundant after training. While feature decomposition layers remain the essential backbone of the network, many other layers can be replaced with zeros without losing performance. Future work may investigate if these layers can be identified and removed during the training phase to save computational resources without sacrificing the final model's efficacy.
Summary updated