Abstract
Residual Networks (ResNets) have become a cornerstone of modern deep learning, primarily due to their ability to facilitate the training of extremely deep architectures. While their empirical success is well-documented, the theoretical underpinnings of their optimization landscape remain an active area of research. This paper investigates the global convergence of Gradient Descent (GD) and Stochastic Gradient Descent (SGD) for training $L$-hidden-layer linear ResNets. We demonstrate that with zero initialization of hidden weights and specific fixed linear transformations at the input and output layers, both GD and SGD converge to the global minimum. Notably, our analysis shows that the network width required for convergence is independent of the network depth $L$, providing a theoretical justification for the scalability of ResNet architectures. Compared to standard deep linear networks, our results offer a sharper width bound by a factor of $O(\kappa L)$, where $\kappa$ is the condition number of the training data covariance matrix.
Introduction
The introduction of skip connections in Residual Networks (ResNets) revolutionized the training of deep neural networks by mitigating the vanishing gradient problem. Despite their ubiquity, the non-convex nature of neural network loss functions makes theoretical guarantees of global convergence difficult to establish. To gain insight into these dynamics, researchers often study deep linear networks. While these models lack non-linear activations, they preserve the complex hierarchical optimization challenges inherent in deep learning.
Previous studies on standard deep linear networks established global convergence but required the network width to scale linearly with the depth $L$. This paper, originally presented at ICLR 2020, addresses whether the ResNet architecture fundamentally relaxes these requirements, potentially explaining why depth is more manageable in residual architectures.
The Linear ResNet Model
We consider an $L$-hidden-layer linear ResNet defined by the function:
$$f_W(x) = B(I + W_L)(I + W_{L-1}) \dots (I + W_1)Ax$$
In this formulation:
- $A$ and $B$ are fixed linear transformations at the input and output layers, respectively.
- $W_l$ represents the weight matrices of the $l$-th hidden layer.
- $(I + W_l)$ represents the skip connection, where $I$ is the identity matrix.
A critical aspect of this study is the use of zero initialization for all hidden weights ($W_l = 0$). This initialization ensures that at the start of training, the network behaves as a simple linear transformation $f_W(x) = BAx$, effectively bypassing the complexities of deep initialization until the weights begin to evolve.
Theoretical Framework
The authors establish global convergence by proving two primary properties of the optimization landscape:
- Restricted Gradient Bounds: The authors show that within a specific region around the initialization, the loss function satisfies a version of the Polyak-Łojasiewicz (PL) condition. This ensures that the gradient remains sufficiently large to guide the weights toward a global minimum as long as they stay within this "well-behaved" region.
- Smoothness and Trajectory Control: The loss function is shown to be sufficiently smooth along the optimization path. This prevents GD and SGD from oscillating or diverging, ensuring that the weights remain within the region where the gradient bounds hold.
Key Findings
1. Depth-Independent Width Requirements
One of the most significant contributions of this work is the proof that the width $m$ required for global convergence does not depend on the number of layers $L$. In standard linear networks, the width requirement grows with depth, but in ResNets, the skip connections decouple the width necessity from the depth of the architecture.
2. Sharper Convergence Bounds
When utilizing Gaussian random transformations for $A$ and $B$, the required width is $O(kr\kappa^2)$. This is sharper than the bounds for standard deep linear networks by a factor of $O(L\kappa)$. This mathematical refinement highlights the efficiency of the residual structure in simplifying the optimization landscape.
3. Modified Identity Transformations
The authors propose a "modified identity" transformation for the fixed layers $A$ and $B$. They prove that a network width of $m \ge d + k$ (where $d$ and $k$ are input and output dimensions) is sufficient to guarantee global convergence. This suggests that the benefits of ResNets are structural and do not rely solely on the randomness of Gaussian initialization.
4. Convergence of SGD
This paper provides the first global convergence rate for SGD in the context of deep linear ResNets. The analysis shows that SGD converges to the global minimum at a rate of $\tilde{O}(\kappa^2 \epsilon^{-1} \log(1/\epsilon) \cdot n/B)$, where $n$ is the number of samples and $B$ is the batch size.
Empirical Validation
The theoretical results were validated using synthetic datasets. The experiments confirmed that while standard gradient descent with identity initialization can encounter optimization hurdles, the ResNet formulation—using either modified identity or random transformations—converges reliably and rapidly to the global minimum.
Conclusion and Future Work
This work provides a rigorous mathematical foundation for the advantages of skip connections. By proving that ResNets allow for depth-independent width requirements and sharper convergence bounds, the authors offer a clear explanation for why these architectures are easier to optimize than their standard counterparts.
While the study is limited to linear networks and fixed input/output layers, the techniques developed—specifically the restricted gradient bounds—provide a roadmap for future analysis of non-linear ResNets. Extending these proofs to architectures with ReLU activations remains a vital next step in bridging the gap between deep learning theory and practice.
Summary updated