ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

Chaoyue Liu, Like Hui

Introduction

In the landscape of deep learning, ReLU nonlinearities are often celebrated for their expressivity—the ability to approximate complex, non-linear functions. However, recent research reveals that these activations provide an additional "free lunch": they fundamentally reshape the gradient landscape to facilitate optimization. By increasing the geometric "angle" between similar data points in the feature space, ReLU activations improve the condition number of the Neural Tangent Kernel (NTK), leading to significantly faster convergence as networks grow deeper.

The Optimization Problem: NTK Conditioning

When training wide neural networks, the Neural Tangent Kernel (NTK) serves as a critical tool for describing how the network evolves during gradient descent. A primary bottleneck in training speed is the condition number ($\kappa$) of this kernel. If the condition number is high—often the case when data points are too "similar" in the gradient space—optimization becomes agonizingly slow.

This research investigates whether the mere presence of ReLU activations improves this condition number compared to a simple linear baseline, and how network depth influences this geometric transformation.

The Separation Phenomenon: ReLU as a Geometric Separator

The core insight of this work is that ReLU acts as a geometric separator. In a standard linear network, the angle between the gradients of two different inputs remains static as the signal propagates through layers. In contrast, ReLU layers effectively "push" similar inputs apart in the gradient feature space.

The model gradient angle $\phi$ converges to a fixed point of ~75.5° as depth $L$ increases in ReLU networks.

As shown in Figure 1, the authors discovered a fascinating mathematical limit: in an infinitely deep ReLU network, any two non-parallel inputs will eventually be separated by a fixed angle of approximately 75.5° in the gradient space, regardless of their initial similarity. This "Universal Angle" ensures that the gradients for different data points become more orthogonal, reducing interference during the learning process.

Empirical Results and Conditioning

The theoretical predictions were validated across diverse benchmarks, including MNIST, FashionMNIST, SVHN, and Librispeech. The results consistently showed that depth, when paired with ReLU, acts as an optimizer rather than a hindrance.

As illustrated in Figure 6, the NTK condition number ($\kappa$) of a ReLU network is strictly smaller than that of its linear counterpart. Furthermore, as network depth ($L$) increases, $\kappa$ decreases monotonically. On the Librispeech dataset, for instance, the condition number dropped by orders of magnitude as depth increased from 1 to 10. This improved conditioning directly translates to faster convergence; deeper ReLU networks reached lower training loss significantly faster than shallower ones across all tested datasets.

Discussion and Limitations

While depth improves optimization, the authors noted a "sweet spot" for generalization. On MNIST, test accuracy peaked at a depth of 6-8 before slightly declining, suggesting that "over-separating" data might eventually discard the similarity information needed to categorize unseen samples.

Additionally, while numerical evidence suggests other functions like GeLU and Tanh exhibit similar separation effects, the rigorous proofs in this study are specific to ReLU. Future work may explore the behavior of these dynamics in narrow "bottleneck" architectures where the infinite-width assumptions of the NTK regime may not fully apply.

Conclusion

This study reframes our understanding of depth and nonlinearity. Rather than just adding complexity, ReLU activations and network depth work in tandem to "soothe" the NTK condition number. By geometrically separating data points, deep ReLU networks create a more well-behaved gradient matrix, providing a robust mechanism for accelerated optimization in wide neural architectures.

Summary updated

BibTeX
@inproceedings{liu2025relu,
  author = {Chaoyue Liu and Like Hui},
  title = {{ReLU soothes the NTK condition number and accelerates optimization for wide neural networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=8dTBydCMSr},
}
Figure