Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$P Parametrization

Zixiang Chen, Greg Yang, Qingyue Zhao, Quanquan Gu

TL;DR

Infinitely wide neural networks have historically forced a theoretical trade-off: they either converge globally without learning features (like the Neural Tangent Kernel) or learn features but suffer from feature collapse in deep architectures (like Mean-Field approaches). This paper proves that Maximal Update Parametrization ($\mu$P) breaks this deadlock, enabling deep, infinitely wide networks to simultaneously learn rich, linearly independent features and guarantee global convergence.

What Problem Does This Paper Address?

Deep learning works incredibly well in practice, but explaining why it successfully converges to optimal solutions despite a highly non-convex optimization landscape remains a major theoretical challenge. To make the math tractable, researchers often study neural networks in the "infinite-width" limit.

The most prominent framework for this is the Neural Tangent Kernel (NTK). While NTK theory successfully proves global convergence, it suffers from a major drawback called "lazy training"—the features learned by the network during training remain essentially identical to their random initialization, failing to capture true representation learning. Conversely, "Mean-Field" frameworks allow for rich feature learning but cause "feature collapse" (where features degenerate into zero vectors) in networks deeper than four layers.

This paper addresses a fundamental question bridging theory and practice: Can deep neural networks simultaneously learn meaningful features and achieve global convergence?

Methods

The authors utilize the Tensor Program (TP) framework to analyze the infinite-width limit of $L$-layer Multilayer Perceptrons (MLPs) under Maximal Update Parametrization ($\mu$P).

To tackle the intricate coupling between forward and backward passes across training iterations, they introduce a two-level filtration framework ($\mathcal{F}_t$ and $\mathcal{G}_t$) to rigorously track the evolution of random variables. They define a class of "GOOD" activation functions (such as Sigmoid, Tanh, and SiLU) that possess smooth, bounded derivatives and prevent degeneracy.

The core mathematical technique involves analyzing the Gaussian processes induced at infinite width. The authors prove that certain second-order invariants (covariance structures) are preserved across adjacent layers during training. This preservation ensures that features remain non-degenerate and linearly independent throughout the entire optimization process.

Datasets

The authors empirically validate their theoretical findings using the CIFAR-10 dataset. Specifically, they train 3-hidden-layer MLPs on a binary classification task using a subset of samples to track feature dynamics across different widths.

Key Insights and Ideas

  • Preserving Feature Richness: Unlike standard or NTK parametrizations where features stay stagnant, $\mu$P allows features to evolve substantially while maintaining their diversity, avoiding the feature collapse seen in deep mean-field models.
  • Linear Independence Guarantees Optimality: By proving that features in each layer remain linearly independent throughout training, the authors demonstrate that the network cannot get trapped in bad local minima. If the training dynamics stabilize, the network is guaranteed to be at a global minimum.
  • Covariance as a Structural Bridge: Even though individual features change dramatically during training, the underlying covariance relationships between adjacent layers remain invariant. This structural consistency allows linear independence to safely propagate forward and backward through deep networks.

Key Results

  • Theoretical Non-Degeneracy: Theorem 4.5 rigorously establishes that both pre-activation and post-activation features remain linearly independent at any time $t$ during training under $\muP$.
  • Global Convergence Verification: Corollary 4.6 shows that when training converges under these conditions, the error signals across mini-batches vanish, proving convergence to a global minimum.
  • Empirical Confirmation: Experiments on CIFAR-10 show that as network width scales up to 4096, only $\mu$P maintains a stable, non-vanishing minimum eigenvalue for the feature Gram matrix, whereas Mean-Field parametrization (IP) experiences catastrophic feature collapse.
Figure 1. Different parametrization schemes exhibit distinct feature learning behaviors as width increases in 3 hidden-layer MLPs. We train it on the CIFAR-10 dataset and measure pre-activation feature properties in the second hidden layer (Layer 2). Left: Feature change (∥h(x) −h0(x)∥2/∥h0(x)∥2) shows only µP maintain stable feature representations. Right: Feature diversity measured by the minimum eigenvalue of the feature gram matrix Kij = ⟨h(xi), h(xj)⟩, where a larger eigenvalue indicates the features span a higher dimensional space. The results reveal that Meanfield parametrization suffers from feature collapse while SP, NTP and µP preserve rich feature representations. Notably, only µP achieves both feature learning capability and feature richness. See Appendix A for experimental details.

Limitations and Future Work

  • Architecture Constraints: The current theoretical proofs are limited to Multilayer Perceptrons (MLPs). Extending this mathematical framework to Transformer architectures (specifically the attention mechanism) and residual networks is a crucial next step.
  • Asymptotic Convergence Rates: While the paper proves that any convergent point is a global minimum, it does not provide explicit bounds on the rate of convergence.
  • The ReLU Gap: The theory strictly requires smooth activation functions with bounded derivatives ("GOOD" functions). Although empirical results in the paper show that $\mu$P also works well with ReLU, explaining the exact mechanism behind ReLU's non-smooth feature learning remains an open theoretical challenge.

Summary updated

Abstract

Despite deep neural networks' powerful representation learning capabilities, theoretical understanding of how networks can simultaneously achieve meaningful feature learning and global convergence remains elusive. Existing approaches like the neural tangent kernel (NTK) are limited because features stay close to their initialization in this parametrization, leaving open questions about feature properties during substantial evolution. In this paper, we investigate the training dynamics of infinitely wide, $L$-layer neural networks using the tensor program (TP) framework. Specifically, we show that, when trained with stochastic gradient descent (SGD) under the Maximal Update parametrization ($μ$P) and mild conditions on the activation function, SGD enables these networks to learn linearly independent features that substantially deviate from their initial values. This rich feature space captures relevant data information and ensures that any convergent point of the training process is a global minimum. Our analysis leverages both the interactions among features across layers and the properties of Gaussian random variables, providing new insights into deep representation learning. We further validate our theoretical findings through experiments on real-world datasets.
BibTeX
@inproceedings{chen2025global,
  author = {Zixiang Chen and Greg Yang and Qingyue Zhao and Quanquan Gu},
  title = {{Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$P Parametrization}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=efWv8jOBkb},
}
Figure