Introduction
In the effort to understand why deep neural networks generalize effectively despite their complexity, researchers often rely on the Neural Tangent Kernel (NTK). The NTK describes the evolution of a network's outputs during training in the infinite-width limit, effectively linearizing the learning process. While the existence of the NTK is well-documented, its internal structure—specifically the eigenfunctions and eigenvalues that define its inductive bias—has remained largely a "black box" for common architectures like Rectified Linear Unit (ReLU) networks. This paper provides a rigorous mathematical map of this structure for 2-layer ReLU networks, identifying the specific functions these networks prefer to learn and quantifying their inherent simplicity bias.
Theoretical Framework and Methodology
The authors analyze a bias-free, 2-layer ReLU network where the hidden layer weights are fixed and drawn from a random Gaussian distribution. This setup corresponds to the "lazy training" or random feature regime. The study focuses on two primary mathematical objects:
- Fisher Information Matrices (FIM): These describe the local geometry of the parameter space.
- Neural Tangent Kernels (NTK): These describe the functional behavior of the network in the output space.
To bridge the gap between these objects, the authors perform an approximate spectral decomposition. By treating the NTK as a linear operator on a Hilbert space ($L^2$ functions over a Gaussian input distribution), they utilize a Taylor-like expansion of the ReLU kernel. This allows them to identify a set of orthonormal eigenfunctions ($F_0, F_l, F_\gamma, F_{\alpha\beta}$) that represent the network's "favorite" functional components.
The Hierarchy of Simplicity
A central finding of the paper is the quantification of the network's inductive bias. The speed at which a network learns a specific component of a target function is directly proportional to the corresponding eigenvalue of the NTK. The authors reveal a clear hierarchy in how information is processed:
- Level 0 (The Norm): The function $F_0(x) \propto |x|$ (the norm of the input) has the largest eigenvalue, meaning the network learns the overall scale of the data fastest.
- Level 1 (Linear Components): Linear terms $F_l(x) = x_l$ are the next priority, with an exact eigenvalue of $1/4$.
- Level 2 (Quadratic Interactions): More complex interactions between input dimensions have significantly smaller eigenvalues, making them much slower to learn.
This hierarchy explains why neural networks exhibit a "simplicity bias," naturally fitting smooth, low-frequency trends before capturing high-frequency "wiggles" or noise.
Key Results and Mathematical Insights
The paper provides explicit analytical formulas for the major eigenvalues, which govern the learning dynamics:
- $\mu_0 \approx \frac{2d+1}{4\pi}$: Dominant eigenvalue for the input norm.
- $\mu_2 \approx \frac{1}{2\pi(d+2)}$: Eigenvalue for quadratic interactions.
Furthermore, the authors demonstrate a striking parallelism between the FIM and the NTK, suggesting that the geometric structure of the weights and the functional behavior of the network are intrinsically linked. For high-dimensional inputs ($d$), the authors prove that their derived eigenfunctions capture the vast majority of the kernel's energy, with the unsolved remainder accounting for less than ~2.6% of the system's total energy.
Discussion and Limitations
While this work provides a concrete "map" for 2-layer ReLU networks, it operates under specific theoretical constraints. The analysis assumes a standard Gaussian input distribution, which, while mathematically tractable, does not capture the manifold-like structure of real-world data. Additionally, the study focuses on the "lazy training" regime where hidden weights remain near their initial random values.
Future research may extend these findings to deeper architectures or regimes where "feature learning" occurs (i.e., where hidden weights adapt significantly to the data). Nevertheless, this paper offers a foundational step toward a more transparent theory of how neural networks perceive and prioritize information based on structural complexity.
Summary updated