Abstract
Deep Residual Networks (ResNets) are foundational to modern computer vision, yet their theoretical behavior at extreme depths remains a subject of intense study. This paper investigates the "degeneration phenomenon" in deep wide ResNets, where the network's learning ability collapses as depth increases if the residual branch scaling remains constant. We demonstrate that by employing a scaling factor $\alpha = L^{-\gamma}$ with $\gamma > 1/2$, the network avoids this collapse and maintains optimal generalization. Our findings, supported by Residual Neural Tangent Kernel (RNTK) analysis and experiments on MNIST and CIFAR datasets, suggest that deep ResNets should be scaled to behave like well-optimized shallow learners to achieve peak performance.
Introduction
Residual Networks (ResNets) revolutionized deep learning by introducing skip connections, allowing for the training of architectures with hundreds or thousands of layers. The standard residual block is defined as $x_{out} = x_{in} + \alpha \cdot f(x_{in})$, where $\alpha$ is a scaling factor. While ResNets are highly effective, a theoretical gap exists in understanding how $\alpha$ should evolve as the network depth $L$ increases. Researchers from Tsinghua University address this by identifying a "degeneration phenomenon" that occurs when $\alpha$ is improperly scaled, leading to poor generalization in very deep networks.
The Degeneration Phenomenon
The core issue identified is that if the scaling factor $\alpha$ remains constant as depth increases ($L \to \infty$), the network's output becomes increasingly insensitive to input features. Mathematically, the Neural Tangent Kernel (NTK)—which characterizes the network's evolution during training—collapses into a constant. When the kernel is constant, the model loses its ability to distinguish between different data points.
As shown in Figure 1, regardless of the initial constant value of $\alpha$, the output converges toward a fixed point as the number of layers increases. This "constant $\alpha$ trap" explains why standard deep ResNets may plateau or degrade in performance without proper architectural adjustments.
Scaling for Optimal Generalization
To prevent this collapse, the authors propose that the residual branch must be scaled down as a function of depth: $\alpha = L^{-\gamma}$. Through spectral and asymptotic analysis of the Residual Neural Tangent Kernel (RNTK), they prove that:
- Slow Decay ($\gamma \leq 1/2$): The degeneration phenomenon still occurs, though at a slower rate.
- Rapid Decay ($\gamma > 1/2$): The infinite-depth RNTK converges to the kernel of a one-hidden-layer fully connected network.
This "shallow deep net" behavior is a key insight: by rapidly scaling down residuals, a deep ResNet retains the learnability of a shallow network while benefiting from the optimization stability of the residual structure. Under this regime, the network can achieve minimax optimal error rates.
Experimental Validation
The theoretical criteria were tested on benchmark datasets including MNIST, CIFAR-10, and CIFAR-100. The experiments compared a constant scaling factor ($\gamma=0$) against the proposed rapid decay ($\gamma=1$).
As illustrated in Figure 2, the model with $\gamma=1$ consistently outperforms the standard $\gamma=0$ configuration. On CIFAR-100 with $L=200$, the properly scaled network shows a significant accuracy boost, reaching approximately 56% compared to the 50% plateau of the constant-scale version. The improvement is even more pronounced as the complexity of the dataset and the depth of the network increase.
Conclusion
This research provides a rigorous mathematical foundation for scaling deep ResNets. By ensuring the residual branch decays at a rate of at least $L^{-1/2}$, practitioners can prevent the degeneration of the neural tangent kernel and ensure that deep networks maintain strong generalization capabilities. Future work will look to extend these findings to non-wide architectures and explore interactions with common normalization techniques like Batch Normalization.
Summary updated