A simple mean field model of feature learning

Niclas Alexander Göring, Chris Mingard, Yoonsoo Nam, Ard A. LouisSubmission to ICLR 2026 · Rejected

TL;DR

While infinite-width theories like the Neural Network Gaussian Process (NNGP) fail to capture feature learning, standard mean-field (MF) approximations capture the onset of learning but vastly underestimate post-transition performance. This paper shows that this failure stems from missing Input Feature Selection (IFS)—a coordinate-level symmetry breaking where networks amplify task-relevant input dimensions. By integrating Automatic Relevance Determination (MF-ARD) into the mean-field prior, the authors derive a tractable model that overcomes the curse of dimensionality and quantitatively predicts the generalization behavior of neural networks trained with Langevin dynamics.

What Problem Does This Paper Address?

In the infinite-width limit, neural networks behave as linear predictors over fixed kernels (NNGP/NTK), incapable of learning internal representations from data. Real-world, finite-width networks, however, dramatically outperform kernel methods on tasks like sparse parity and index models because they learn representations ("feature learning").

Existing theoretical attempts to model feature learning fall into two extremes:

  1. Dynamical mean-field theories (DMFT), which track representations via complicated integro-differential equations during training.
  2. Static Bayesian/posterior theories, which analyze networks trained via Stochastic Gradient Langevin Dynamics (SGLD) at stationarity.

While basic static mean-field approximations capture the onset of feature learning as a symmetry-breaking phase transition, they severely underestimate generalization gains once learning begins. The authors investigate why plain mean-field theory fails and how to fix it without sacrificing analytical tractability.

Methods

The authors analyze two-layer non-linear neural networks trained with SGLD, which samples from a Gibbs posterior distribution:

  1. Mean-Field (MF) Approximation: SGLD induces dense neuron–neuron coupling via a Gram interaction kernel $G(w_i, w_j)$. Standard mean-field decouples this interacting system into independent neurons interacting only with an effective, collective "mean field" $\langle f(x) \rangle$.
  2. Diagnosing the Homogeneity Gap: Plain MF assumes an isotropic prior over weight coordinates, imposing the same regularizing shrinkage along all input directions. In real networks, however, gradients along task-relevant coordinates accumulate constructively while noise coordinates average out.
  3. MF with Automatic Relevance Determination (MF-ARD): To capture this coordinate-specific feedback, the authors endow the weight prior with learnable, per-coordinate precisions (stiffness parameters) $\rho_j$. Using SGLD stationarity and linear response theory, they derive self-consistent fixed-point updates: $$\rho_j \propto \left(\beta_0 + \frac{N}{2}\langle w_j^2 \rangle\right)^{-1}$$ When a coordinate aligns with the target, its variance increases, reducing $\rho_j$ and relaxing regularization. Irrelevant coordinates maintain high $\rho_j$, keeping them tightly confined to zero.

  4. Iterative Streaming Algorithm: The fixed-point equations are solved numerically using an ensemble particle method that streams data mini-statistics, preserving finite-sample ($P$) fluctuations.

Datasets

The authors evaluate their theory on synthetic benchmark tasks where kernel methods are known to suffer from the curse of dimensionality:

  • $k$-Sparse Parity: Inputs on the Boolean hypercube $x \in \{\pm 1\}^d$ with teacher $y = \prod_{j \in S} x_j$ for support size $|S| = k = 4$ in ambient dimension $d = 35$.
  • Single-Index Hermite Model: Inputs $x \sim \mathcal{N}(0, I_d)$ with teacher $y = \mathrm{He}_4(w^\top x)$, where $w$ has support $k=2$ in $d=18$, passed through a 4th-degree Hermite polynomial.

Key Insights and Ideas

  • Feature Learning as a Two-Stage Process:
  • Onset (Phase Transition): When sample size $P$ exceeds a critical threshold $P_c$ (or noise $\kappa < \kappa_c$), the trivial isotropic state becomes unstable, breaking symmetry toward the target.
  • Specialization via IFS: A self-reinforcing dynamic where well-aligned neurons reduce the collective residual, shutting off gradient signals for unaligned dimensions and causing heavy-tailed coordinate distributions.
  • Beating the Curse of Dimensionality: Theorem 4.1 proves that the critical noise threshold scales as $\kappa_c^2 \asymp \Theta(\sqrt{1/(dk)})$ for standard MF, which suffers an ambient dimension penalty $d$. For MF-ARD, this threshold improves to $\Theta(\sqrt{1/k})$, depending strictly on the intrinsic target dimension $k$.
  • Intra-Neuron vs. Inter-Neuron Sparsity: Unlike prior work studying neuron-wise sparsity (how many neurons activate), this work shows that intra-neuron coordinate anisotropy is the causal mechanism driving feature learning advantages over kernels. Enforcing coordinate homogeneity with a penalty degrades network performance back to the NNGP kernel baseline.

Key Results

  • Quantitative Generalization Prediction: MF-ARD closely tracks the test mean squared error (MSE) learning curves of SGLD-trained networks across dataset sizes ($P \in [10, 10^4]$), whereas plain MF overestimates error post-transition by orders of magnitude.
  • Phase Diagram Accuracy: On $(P, \kappa)$ phase diagrams, MF-ARD reproduces the sharp boundary seen in SGLD, including the subtle "helpful noise" regime where moderate noise lowers the critical sample complexity $P_c$.
  • Distribution Marginals: Weight marginal distributions $p(w_j)$ under MF-ARD exhibit the exact heavy-tailed non-Gaussian behavior on relevant features observed in full network simulations.

Limitations and Future Work

  • Architecture Scope: Analysis is currently restricted to shallow (two-layer) fully connected networks. Extending the framework to deep networks, convolutional layers, or attention mechanisms remains open.
  • Task Specificity: The ARD formulation is especially well-suited for axis-aligned, sparse coordinate tasks ($k$-sparse parity, sparse index models). Settings requiring dense, distributed, or smooth representations may require generalized covariance adaptations rather than diagonal coordinate precisions.
  • Static vs. Dynamical Gap: MF-ARD is a static equilibrium theory of the posterior; directly mapping these fixed points onto non-equilibrium gradient descent trajectories is left for future research.

Summary updated

Abstract

Feature learning (FL), where neural networks adapt their internal representations during training, remains poorly understood. Using methods from statistical physics, we derive a tractable, self-consistent mean-field (MF) theory for the Bayesian posterior of two-layer non-linear networks trained with stochastic gradient Langevin dynamics (SGLD). At infinite width, this theory reduces to kernel ridge regression, but at finite width it predicts a symmetry breaking phase transition where networks abruptly align with target functions. While the basic MF theory provides theoretical insight into the emergence of FL in the finite-width regime, semi-quantitatively predicting the onset of FL with noise or sample size, it substantially underestimates the improvements in generalisation after the transition. We trace this discrepancy to a key mechanism absent from the plain MF description: \textit{self-reinforcing input feature selection}. Incorporating this mechanism into the MF theory allows us to quantitatively match the learning curves of SGLD-trained networks and provides mechanistic insight into FL.
BibTeX
@misc{goring2025simple,
  author = {Niclas Alexander Göring and Chris Mingard and Yoonsoo Nam and Ard A. Louis},
  title = {{A simple mean field model of feature learning}},
  year = {2025},
  eprint = {2510.15174},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.15174},
}
Figure