Abstract
This research establishes a new, tighter theoretical generalization bound for Residual Networks (ResNets) using $\ell_{p,q}$-norm weight normalization. By applying Rademacher Complexity analysis, the study demonstrates that ResNet architectures inherently possess a lower generalization error bound than standard Deep Neural Networks (DNNs), providing a mathematical foundation for the superior performance of "shortcut" connections in deep learning.
Introduction
In deep learning, "generalization" refers to a model's ability to perform well on unseen data. While ResNets are highly successful in practice, explaining their generalization capabilities from a theoretical standpoint remains challenging. Most existing bounds are designed for simple DNNs, often ignoring practical details like bias terms or specific architectural shortcuts. Furthermore, many previous bounds depend heavily on the "width" (number of neurons) of the network, which contradicts the empirical success of very wide networks. This paper derives a "width-independent" and tighter bound specifically for the ResNet family to better understand their efficiency and robustness.
Theoretical Framework
The researchers utilize Rademacher Complexity to quantify how well a function class can fit random noise. Their approach incorporates several key components:
- Norm-Based Capacity Control: The study employs $\ell_{p,q}$-norm weight normalization. By assuming weights are constrained within certain norms (specifically where $1/p + 1/q \ge 1$), the authors can effectively control the network's complexity.
- Realistic Modeling: Unlike many theoretical works that simplify architectures, this study explicitly includes bias terms and layer-wise normalization in the mathematical formulation of residual blocks.
- Width-Independent Bounds: The derived bound relies on the depth of the network (via a square root term) but remains independent of the width of the hidden layers, explaining why wide ResNets generalize well without immediate overfitting.
- Architectural Extensions: The framework extends to standard DNNs and Convolutional Neural Networks (CNNs), treating convolutions as a special case of sparse linear connections.
Experimental Validation
To verify the theoretical findings, the authors conducted numerical experiments using a synthetic non-linear regression task. The dataset consisted of 500 training samples and 1500 test samples in a 300-dimensional space. The study compared two specific models: Net-A (a standard 4-layer DNN) and Net-B (the same DNN but with a residual shortcut between the second and third layers).
Results and Discussion
The most significant insight is that residual connections (the "identity" path) mathematically reduce the generalization bound. The derived bound for ResNets is $O\left(\gamma c_k \prod_{l=1}^{k-1} (c_l + 1) \sqrt{\frac{k}{n}}\right)$, which is tighter than previous bounds that included more complex logarithmic and product terms.
In simulations, even when the test errors of the DNN and ResNet were similar, the calculated generalization bound for the ResNet was consistently lower. In over 95% of experimental trials, the ResNet exhibited superior generalization properties. For practitioners, the paper suggests that to avoid overfitting, one should ensure $1/p + 1/q \ge 1$ for weight norms, keep layer-wise norm constants ($c_i$) small, and prefer residual structures over plain ones.
Conclusion
While this work provides a tighter bound and mathematical justification for ResNets, the authors acknowledge that the numerical values of these bounds can still be large (a common issue in deep learning theory). Future work will focus on generalization bounds within smaller function spaces, such as the vicinity of local minima reached during training, and developing optimization algorithms that explicitly minimize these weight-normalized bounds.
Summary updated