Abstract
Despite deep neural networks' powerful representation learning capabilities, theoretical understanding of how networks can simultaneously achieve meaningful feature learning and global convergence remains elusive. Existing approaches like the neural tangent kernel (NTK) are limited because features stay close to their initialization in this parametrization, leaving open questions about feature properties during substantial evolution. In this paper, we investigate the training dynamics of infinitely wide, $L$-layer neural networks using the tensor program (TP) framework. Specifically, we show that, when trained with stochastic gradient descent (SGD) under the Maximal Update parametrization ($\mu$P) and mild conditions on the activation function, SGD enables these networks to learn linearly independent features that substantially deviate from their initial values. This rich feature space captures relevant data information and ensures that any convergent point of the training process is a global minimum. Our analysis leverages both the interactions among features across layers and the properties of Gaussian random variables, providing new insights into deep representation learning. We further validate our theoretical findings through experiments on real-world datasets.
Introduction
A fundamental tension exists in deep learning theory between the ability of a network to learn complex representations and the mathematical guarantee that it will converge to an optimal solution. While empirical success suggests that deep networks excel at both, theoretical frameworks often capture only one side of the coin. The Neural Tangent Kernel (NTK) regime provides convergence guarantees but assumes "lazy training," where features remain static. Conversely, Mean Field regimes allow for "rich" feature evolution but often encounter "feature collapse" in deep architectures, where gradients vanish or representations become redundant.
Presented at ICML 2025, this research provides a rigorous foundation for Maximal Update Parametrization ($\mu$P), proving that it is possible to achieve both rich feature learning and global convergence in deep, infinitely wide neural networks.
The Theoretical Divide: NTK vs. Mean Field
To understand the significance of this work, one must look at the two dominant paradigms in infinite-width analysis:
- Neural Tangent Kernel (NTK): In this regime, the network behaves like a linear model around its initialization. While this allows for proofs of global convergence, it fails to model how networks actually learn features from data.
- Mean Field / Integrable Parametrization (IP): This allows features to evolve significantly. However, in deep networks (typically beyond 4 layers), these models often suffer from degeneracy. Features across different layers can become linearly dependent or the signal can vanish, preventing effective training.
The authors address a critical question: Can a deep network evolve its features substantially while maintaining the mathematical properties needed to guarantee a global optimum?
Methodology: Tensor Programs and $\mu$P
The study utilizes the Tensor Program (TP) framework to analyze the behavior of $L$-layer networks as their width ($n$) approaches infinity. The core of the analysis is the Maximal Update Parametrization ($\mu$P). Unlike standard parametrization, $\mu$P scales initialization variances and learning rates such that every layer—regardless of depth—undergoes a non-trivial update in the infinite-width limit.
The researchers employ an inductive proof strategy using a two-level filtration system ($\mathcal{F}_t$ and $\mathcal{G}_t$) to track the evolution of activations and gradients across layers and training steps. A key requirement for their proof is the use of "GOOD" activation functions—such as SiLU, Tanh, and GeLU—which are smooth and possess non-collapsing properties that prevent feature degeneracy.
Key Theoretical Contributions
The paper presents two landmark results regarding the training dynamics under $\mu$P:
1. Feature Non-Degeneracy (Theorem 4.5)
The authors prove that under $\mu$P, both pre-activation and post-activation features remain linearly independent throughout the training process. This "richness" ensures that the network maintains a high-dimensional representation of the input data, avoiding the feature collapse seen in other parametrizations. This is achieved by showing that the training induces stable Gaussian processes for both forward and backward passes.
2. Global Convergence (Corollary 4.6)
Because the features remain non-degenerate and linearly independent, the network always maintains a viable gradient path. The authors demonstrate that if the training process converges, it is mathematically guaranteed to reach a global minimum where the training error is zero. This bridges the gap between the convergence guarantees of NTK and the representation learning capabilities of Mean Field theories.
Empirical Validation
The theoretical findings were validated using Multi-Layer Perceptrons (MLPs) trained on the CIFAR-10 dataset. The experiments compared $\mu$P against Standard Parametrization (SP), Neural Tangent Parametrization (NTP), and Integrable Parametrization (IP/Mean Field). The results confirmed that $\mu$P is uniquely capable of maintaining high feature diversity (measured by the minimum eigenvalue of the feature Gram matrix) while allowing for significant feature evolution as the network width increases.
Discussion and Future Directions
While this paper provides a major step forward in understanding deep representation learning, several avenues for future research remain:
- Architectural Expansion: The current proof focuses on MLPs. Extending these non-degeneracy guarantees to Transformers and the attention mechanism is a critical next step for modern AI.
- Convergence Rates: While the paper proves that global convergence occurs, the specific rate of convergence under $\mu$P remains to be quantified.
- Non-Smooth Activations: The current theory requires twice continuously differentiable activation functions. While $\mu$P performs well with ReLU in practice, a formal proof for non-smooth functions is still needed.
By reconciling feature learning with global convergence, this work provides a robust theoretical justification for the scaling laws and parametrization strategies currently driving the state-of-the-art in deep learning.
Summary updated