Introduction
In deep learning theory, studying neural networks as their width approaches infinity has led to massive breakthroughs, including the Neural Network-Gaussian Process (NNGP) correspondence and the Neural Tangent Kernel (NTK). Recently, a framework called Tensor Programs (TP) automated the calculation of these infinite-width limits for virtually any network architecture. This framework yielded practical tools like Maximal Update Parametrization ($\mu$P), which allows researchers to tune hyperparameters on small models and transfer them zero-shot to massive models like GPT-3.
However, a significant theoretical gap remained: the foundational "Master Theorem" of Tensor Programs strictly assumed that a network's weights were initialized from a Gaussian distribution. In practice, engineers frequently use uniform or truncated Gaussian distributions to avoid numerical instabilities caused by extreme outliers. While theorists often assume "universality"—the idea that the exact distribution does not matter at a macroscopic scale—this had never been rigorously proven for the complex computational graphs of modern neural networks. This paper closes that gap by developing a comprehensive theory of non-Gaussian Tensor Programs.
Parameter Categorization and Methods
To address the non-Gaussian initialization problem, the authors introduce a precise categorization of neural network parameters based on how their dimensions scale as the network's width ($n$) grows:
- Scalar-like: Neither dimension grows with $n$ (e.g., the bias of a single output neuron).
- Vector-like: Exactly one dimension grows with $n$ (e.g., input layer weights, output layer weights, or hidden layer biases).
- Matrix-like: Both dimensions grow with $n$ (e.g., hidden-to-hidden layer weight matrices).
Proving the new Non-Gaussian Master Theorem required departing from the standard Gaussian conditioning tricks used in previous TP literature. Instead, the authors employ a sophisticated mathematical interpolation technique. They construct an "interpolated program" that smoothly transitions the network's matrices from a pure Gaussian distribution ($t=0$) to the target non-Gaussian distribution ($t=1$) using a specific trigonometric formula. By proving that the derivatives of the program's outputs with respect to this transition variable $t$ vanish as the network becomes infinitely wide, they demonstrate that the final macroscopic behavior remains identical.
The Universality Principle of Initialization
The core insight of the paper is the Universality Principle of Initialization. The authors prove that the exact probability distribution used to initialize weights does not matter for matrix-like parameters, provided the mean and variance remain consistent.
However, the distribution does matter for vector-like and scalar-like parameters. If a practitioner swaps a Gaussian initialization for a uniform one (with the same variance) on the hidden layers, a wide network will train exactly the same way. But if the same swap is applied to the input or output layers, the network's initial outputs and learning dynamics will fundamentally change.
Crucially, this principle mathematically justifies why modern hyperparameter transfer techniques like $\mu$P are distributionally universal. The $\mu$Transfer technique remains mathematically sound—and continues to save immense compute costs—even if the network is initialized with uniform or truncated Gaussian weights.
Key Theoretical Results
As a purely theoretical machine learning paper, the core contributions are mathematical proofs that extend the boundaries of infinite-width neural network analysis:
- Non-Gaussian Master Theorem (Theorem 3.7): The authors rigorously prove that as network width $n \to \infty$, the scalars in a Tensor Program converge to the exact same deterministic limits whether the matrices are Gaussian or non-Gaussian (provided the non-Gaussian matrices have zero mean, variance $1/n$, and bounded higher moments).
- NNGP and NTK Extensions: The paper successfully extends the NNGP correspondence and NTK training dynamics to networks with non-Gaussian weight initializations.
- Random Matrix Theory: The framework automatically generalizes classical random matrix theory results, such as the Semicircle Law and Marchenko-Pastur Law, to non-Gaussian ensembles within the context of neural network computation graphs.
Limitations and Future Directions
The primary limitation of the current proof technique for the Non-Gaussian Master Theorem is its strict smoothness requirement. The network's activation functions must be polynomially smooth, which strictly rules out non-smooth functions like pure ReLU or MaxPool.
As a practical workaround, the authors note that practitioners can use smoothed versions of these functions, such as Softplus with a very small temperature parameter, to approximate ReLU and satisfy the theorem's constraints. For future work, the authors suggest that the $L^p$ convergence established in their theorem could likely be proven without requiring such strict smoothness assumptions, leaving this as an open challenge for theoretical machine learning.
Summary updated