Improve Generalization Ability of Deep Wide Residual Network with A Suitable Scaling Factor

Songtao Tian, Zixiong Yu

Abstract

Deep Residual Networks (ResNets) are foundational to modern computer vision, yet their theoretical behavior at extreme depths remains a subject of intense study. This paper investigates the "degeneration phenomenon" in deep wide ResNets, where the network's learning ability collapses as depth increases if the residual branch scaling remains constant. We demonstrate that by employing a scaling factor $\alpha = L^{-\gamma}$ with $\gamma > 1/2$, the network avoids this collapse and maintains optimal generalization. Our findings, supported by Residual Neural Tangent Kernel (RNTK) analysis and experiments on MNIST and CIFAR datasets, suggest that deep ResNets should be scaled to behave like well-optimized shallow learners to achieve peak performance.

Introduction

Residual Networks (ResNets) revolutionized deep learning by introducing skip connections, allowing for the training of architectures with hundreds or thousands of layers. The standard residual block is defined as $x_{out} = x_{in} + \alpha \cdot f(x_{in})$, where $\alpha$ is a scaling factor. While ResNets are highly effective, a theoretical gap exists in understanding how $\alpha$ should evolve as the network depth $L$ increases. Researchers from Tsinghua University address this by identifying a "degeneration phenomenon" that occurs when $\alpha$ is improperly scaled, leading to poor generalization in very deep networks.

The Degeneration Phenomenon

The core issue identified is that if the scaling factor $\alpha$ remains constant as depth increases ($L \to \infty$), the network's output becomes increasingly insensitive to input features. Mathematically, the Neural Tangent Kernel (NTK)—which characterizes the network's evolution during training—collapses into a constant. When the kernel is constant, the model loses its ability to distinguish between different data points.

Figure 1: The degeneration of network output as depth L increases for various constant scaling factors alpha. As L grows, the output converges to a constant value, indicating a loss of input sensitivity.

As shown in Figure 1, regardless of the initial constant value of $\alpha$, the output converges toward a fixed point as the number of layers increases. This "constant $\alpha$ trap" explains why standard deep ResNets may plateau or degrade in performance without proper architectural adjustments.

Scaling for Optimal Generalization

To prevent this collapse, the authors propose that the residual branch must be scaled down as a function of depth: $\alpha = L^{-\gamma}$. Through spectral and asymptotic analysis of the Residual Neural Tangent Kernel (RNTK), they prove that:

  • Slow Decay ($\gamma \leq 1/2$): The degeneration phenomenon still occurs, though at a slower rate.
  • Rapid Decay ($\gamma > 1/2$): The infinite-depth RNTK converges to the kernel of a one-hidden-layer fully connected network.

This "shallow deep net" behavior is a key insight: by rapidly scaling down residuals, a deep ResNet retains the learnability of a shallow network while benefiting from the optimization stability of the residual structure. Under this regime, the network can achieve minimax optimal error rates.

Experimental Validation

The theoretical criteria were tested on benchmark datasets including MNIST, CIFAR-10, and CIFAR-100. The experiments compared a constant scaling factor ($\gamma=0$) against the proposed rapid decay ($\gamma=1$).

Figure 2: Test accuracy comparison between constant scaling (gamma=0) and rapid decay scaling (gamma=1) on CIFAR-10 and CIFAR-100 datasets.

As illustrated in Figure 2, the model with $\gamma=1$ consistently outperforms the standard $\gamma=0$ configuration. On CIFAR-100 with $L=200$, the properly scaled network shows a significant accuracy boost, reaching approximately 56% compared to the 50% plateau of the constant-scale version. The improvement is even more pronounced as the complexity of the dataset and the depth of the network increase.

Conclusion

This research provides a rigorous mathematical foundation for scaling deep ResNets. By ensuring the residual branch decays at a rate of at least $L^{-1/2}$, practitioners can prevent the degeneration of the neural tangent kernel and ensure that deep networks maintain strong generalization capabilities. Future work will look to extend these findings to non-wide architectures and explore interactions with common normalization techniques like Batch Normalization.

Summary updated

BibTeX
@inproceedings{tian2026improve,
  author = {Songtao Tian and Zixiong Yu},
  title = {{Improve Generalization Ability of Deep Wide Residual Network with A Suitable Scaling Factor}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=iWV5o07Bxv},
}
Figure