Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escape

Ioannis Bantzis, James B. Simon, Arthur Jacot

Introduction

Understanding the training dynamics of deep neural networks remains a fundamental challenge in machine learning. While the "kernel" or "lazy" regime—where networks behave like fixed linear models—is well-understood, the "active" or "rich" regime, where networks learn complex features, is more elusive. This paper investigates the active regime by examining how deep ReLU networks escape the initial saddle point at the origin when initialized with vanishingly small weights.

The Problem: Escaping the Origin

When weights are initialized near zero, Gradient Descent (GD) often encounters a saddle point at the origin ($\theta = 0$). The network must find an "escape direction" to reduce loss and progress toward a solution. While this "saddle-to-saddle" behavior has been characterized for linear networks, its manifestation in non-linear ReLU networks was previously unclear. This research aims to define the structure of these first escape directions in deep ReLU models.

Methodology

The authors utilize a combination of theoretical derivation and empirical testing:

  • Localized Loss Analysis: By treating the loss as a homogeneous function of degree $L$ (the network depth) near the origin, they decompose dynamics into weight norm and direction.
  • Mathematical Rigor: Theorem 3.1 provides a formal bound on the rank of weight matrices during the initial escape.
  • Empirical Validation: Experiments were conducted using 4-layer and 6-layer fully connected networks on the MNIST dataset and a synthetic "unit circle" dataset.

Key Insights: The Half-Bottleneck and Depth

A primary discovery is the "half-bottleneck" structure. While initial layers may exhibit higher-rank configurations, deeper layers (closer to the output) are increasingly biased toward rank-1 transformations during the first escape.

Interestingly, depth appears to increase linearity in this phase; the influence of the ReLU nonlinearity weakens as one moves deeper into the network. Furthermore, the "speed" of escape from the origin saddle point increases with depth, suggesting that deeper architectures may inherently facilitate the early stages of feature learning.

Key Results and Empirical Evidence

Theorem 3.1 establishes that for a layer $\ell$, the ratio of "tail" singular values to the primary singular value is bounded by $O(\ell^{-1/2})$. This implies that as depth increases, the weight matrix $W_\ell$ mathematically converges toward a rank-1 structure.

Experiments on MNIST confirm these predictions. Immediately following the first significant drop in training loss, a single dominant singular value emerges across all layers, a phenomenon most pronounced in the final layers of the network.

Training loss and singular value distributions during the first saddle escape on MNIST.

The study also notes that rank-1 is a bias rather than an absolute rule. In certain datasets, such as the synthetic unit circle, the first layers must adopt a higher rank (e.g., rank-2) to effectively fit the data, though the low-rank bias persists in deeper layers.

Limitations and Future Directions

This research focuses specifically on the first escape from the origin under very small initialization. Whether these findings generalize to standard initialization schemes (e.g., He or Xavier) or more complex architectures like Transformers and ResNets remains an open question. Future work is needed to prove that these approximately low-rank layers persist long enough to reach subsequent saddle points, potentially completing the theoretical framework for incremental learning in deep networks.

Summary updated

BibTeX
@inproceedings{bantzis2026saddle,
  author = {Ioannis Bantzis and James B. Simon and Arthur Jacot},
  title = {{Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escape}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=B4zcoLvjw0},
}
Figure