Theoretical Investigation of Generalization Bound for Residual Networks

Hao Chen, Zhanfeng Mo, Zhouwang Yang, Xiao Wang

Abstract

This research establishes a new, tighter theoretical generalization bound for Residual Networks (ResNets) using $\ell_{p,q}$-norm weight normalization. By applying Rademacher Complexity analysis, the study demonstrates that ResNet architectures inherently possess a lower generalization error bound than standard Deep Neural Networks (DNNs), providing a mathematical foundation for the superior performance of "shortcut" connections in deep learning.

Introduction

In deep learning, "generalization" refers to a model's ability to perform well on unseen data. While ResNets are highly successful in practice, explaining their generalization capabilities from a theoretical standpoint remains challenging. Most existing bounds are designed for simple DNNs, often ignoring practical details like bias terms or specific architectural shortcuts. Furthermore, many previous bounds depend heavily on the "width" (number of neurons) of the network, which contradicts the empirical success of very wide networks. This paper derives a "width-independent" and tighter bound specifically for the ResNet family to better understand their efficiency and robustness.

Theoretical Framework

The researchers utilize Rademacher Complexity to quantify how well a function class can fit random noise. Their approach incorporates several key components:

  • Norm-Based Capacity Control: The study employs $\ell_{p,q}$-norm weight normalization. By assuming weights are constrained within certain norms (specifically where $1/p + 1/q \ge 1$), the authors can effectively control the network's complexity.
  • Realistic Modeling: Unlike many theoretical works that simplify architectures, this study explicitly includes bias terms and layer-wise normalization in the mathematical formulation of residual blocks.
  • Width-Independent Bounds: The derived bound relies on the depth of the network (via a square root term) but remains independent of the width of the hidden layers, explaining why wide ResNets generalize well without immediate overfitting.
  • Architectural Extensions: The framework extends to standard DNNs and Convolutional Neural Networks (CNNs), treating convolutions as a special case of sparse linear connections.

Experimental Validation

To verify the theoretical findings, the authors conducted numerical experiments using a synthetic non-linear regression task. The dataset consisted of 500 training samples and 1500 test samples in a 300-dimensional space. The study compared two specific models: Net-A (a standard 4-layer DNN) and Net-B (the same DNN but with a residual shortcut between the second and third layers).

Figure 3: Architecture comparison of Net-A (plain DNN) and Net-B (ResNet) used in the experiments.

Results and Discussion

The most significant insight is that residual connections (the "identity" path) mathematically reduce the generalization bound. The derived bound for ResNets is $O\left(\gamma c_k \prod_{l=1}^{k-1} (c_l + 1) \sqrt{\frac{k}{n}}\right)$, which is tighter than previous bounds that included more complex logarithmic and product terms.

Figure 2: Empirical comparison showing that Net-B (ResNet) consistently achieves a lower generalization bound than Net-A (DNN), despite similar test errors.

In simulations, even when the test errors of the DNN and ResNet were similar, the calculated generalization bound for the ResNet was consistently lower. In over 95% of experimental trials, the ResNet exhibited superior generalization properties. For practitioners, the paper suggests that to avoid overfitting, one should ensure $1/p + 1/q \ge 1$ for weight norms, keep layer-wise norm constants ($c_i$) small, and prefer residual structures over plain ones.

Conclusion

While this work provides a tighter bound and mathematical justification for ResNets, the authors acknowledge that the numerical values of these bounds can still be large (a common issue in deep learning theory). Future work will focus on generalization bounds within smaller function spaces, such as the vicinity of local minima reached during training, and developing optimization algorithms that explicitly minimize these weight-normalized bounds.

Summary updated

Abstract

This paper presents a framework for norm-based capacity control with respect to an lp,q-norm in weight-normalized Residual Neural Networks (ResNets). We first formulate the representation of each residual block. For the regression problem, we analyze the Rademacher Complexity of the ResNets family. We also establish a tighter generalization upper bound for weight-normalized ResNets. in a more general sight. Using the lp,q-norm weight normalization in which 1/p+1/q >=1, we discuss the properties of a width-independent capacity control, which only relies on the depth according to a square root term. Several comparisons suggest that our result is tighter than previous work. Parallel results for Deep Neural Networks (DNN) and Convolutional Neural Networks (CNN) are included by introducing the lp,q-norm weight normalization for DNN and the lp,q-norm kernel normalization for CNN. Numerical experiments also verify that ResNet structures contribute to better generalization properties.
BibTeX
@inproceedings{chen2019theoretical,
  author = {Hao Chen and Zhanfeng Mo and Zhouwang Yang and Xiao Wang},
  title = {{Theoretical Investigation of Generalization Bound for Residual Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/288},
}
Figure