TL;DR
While infinite-width theories like the Neural Network Gaussian Process (NNGP) fail to capture feature learning, standard mean-field (MF) approximations capture the onset of learning but vastly underestimate post-transition performance. This paper shows that this failure stems from missing Input Feature Selection (IFS)—a coordinate-level symmetry breaking where networks amplify task-relevant input dimensions. By integrating Automatic Relevance Determination (MF-ARD) into the mean-field prior, the authors derive a tractable model that overcomes the curse of dimensionality and quantitatively predicts the generalization behavior of neural networks trained with Langevin dynamics.
What Problem Does This Paper Address?
In the infinite-width limit, neural networks behave as linear predictors over fixed kernels (NNGP/NTK), incapable of learning internal representations from data. Real-world, finite-width networks, however, dramatically outperform kernel methods on tasks like sparse parity and index models because they learn representations ("feature learning").
Existing theoretical attempts to model feature learning fall into two extremes:
- Dynamical mean-field theories (DMFT), which track representations via complicated integro-differential equations during training.
- Static Bayesian/posterior theories, which analyze networks trained via Stochastic Gradient Langevin Dynamics (SGLD) at stationarity.
While basic static mean-field approximations capture the onset of feature learning as a symmetry-breaking phase transition, they severely underestimate generalization gains once learning begins. The authors investigate why plain mean-field theory fails and how to fix it without sacrificing analytical tractability.
Methods
The authors analyze two-layer non-linear neural networks trained with SGLD, which samples from a Gibbs posterior distribution:
- Mean-Field (MF) Approximation: SGLD induces dense neuron–neuron coupling via a Gram interaction kernel $G(w_i, w_j)$. Standard mean-field decouples this interacting system into independent neurons interacting only with an effective, collective "mean field" $\langle f(x) \rangle$.
- Diagnosing the Homogeneity Gap: Plain MF assumes an isotropic prior over weight coordinates, imposing the same regularizing shrinkage along all input directions. In real networks, however, gradients along task-relevant coordinates accumulate constructively while noise coordinates average out.
-
MF with Automatic Relevance Determination (MF-ARD): To capture this coordinate-specific feedback, the authors endow the weight prior with learnable, per-coordinate precisions (stiffness parameters) $\rho_j$. Using SGLD stationarity and linear response theory, they derive self-consistent fixed-point updates: $$\rho_j \propto \left(\beta_0 + \frac{N}{2}\langle w_j^2 \rangle\right)^{-1}$$ When a coordinate aligns with the target, its variance increases, reducing $\rho_j$ and relaxing regularization. Irrelevant coordinates maintain high $\rho_j$, keeping them tightly confined to zero.
-
Iterative Streaming Algorithm: The fixed-point equations are solved numerically using an ensemble particle method that streams data mini-statistics, preserving finite-sample ($P$) fluctuations.
Datasets
The authors evaluate their theory on synthetic benchmark tasks where kernel methods are known to suffer from the curse of dimensionality:
- $k$-Sparse Parity: Inputs on the Boolean hypercube $x \in \{\pm 1\}^d$ with teacher $y = \prod_{j \in S} x_j$ for support size $|S| = k = 4$ in ambient dimension $d = 35$.
- Single-Index Hermite Model: Inputs $x \sim \mathcal{N}(0, I_d)$ with teacher $y = \mathrm{He}_4(w^\top x)$, where $w$ has support $k=2$ in $d=18$, passed through a 4th-degree Hermite polynomial.
Key Insights and Ideas
- Feature Learning as a Two-Stage Process:
- Onset (Phase Transition): When sample size $P$ exceeds a critical threshold $P_c$ (or noise $\kappa < \kappa_c$), the trivial isotropic state becomes unstable, breaking symmetry toward the target.
- Specialization via IFS: A self-reinforcing dynamic where well-aligned neurons reduce the collective residual, shutting off gradient signals for unaligned dimensions and causing heavy-tailed coordinate distributions.
- Beating the Curse of Dimensionality: Theorem 4.1 proves that the critical noise threshold scales as $\kappa_c^2 \asymp \Theta(\sqrt{1/(dk)})$ for standard MF, which suffers an ambient dimension penalty $d$. For MF-ARD, this threshold improves to $\Theta(\sqrt{1/k})$, depending strictly on the intrinsic target dimension $k$.
- Intra-Neuron vs. Inter-Neuron Sparsity: Unlike prior work studying neuron-wise sparsity (how many neurons activate), this work shows that intra-neuron coordinate anisotropy is the causal mechanism driving feature learning advantages over kernels. Enforcing coordinate homogeneity with a penalty degrades network performance back to the NNGP kernel baseline.
Key Results
- Quantitative Generalization Prediction: MF-ARD closely tracks the test mean squared error (MSE) learning curves of SGLD-trained networks across dataset sizes ($P \in [10, 10^4]$), whereas plain MF overestimates error post-transition by orders of magnitude.
- Phase Diagram Accuracy: On $(P, \kappa)$ phase diagrams, MF-ARD reproduces the sharp boundary seen in SGLD, including the subtle "helpful noise" regime where moderate noise lowers the critical sample complexity $P_c$.
- Distribution Marginals: Weight marginal distributions $p(w_j)$ under MF-ARD exhibit the exact heavy-tailed non-Gaussian behavior on relevant features observed in full network simulations.
Limitations and Future Work
- Architecture Scope: Analysis is currently restricted to shallow (two-layer) fully connected networks. Extending the framework to deep networks, convolutional layers, or attention mechanisms remains open.
- Task Specificity: The ARD formulation is especially well-suited for axis-aligned, sparse coordinate tasks ($k$-sparse parity, sparse index models). Settings requiring dense, distributed, or smooth representations may require generalized covariance adaptations rather than diagonal coordinate precisions.
- Static vs. Dynamical Gap: MF-ARD is a static equilibrium theory of the posterior; directly mapping these fixed points onto non-equilibrium gradient descent trajectories is left for future research.
Summary updated