A simple mean field model of feature learning

Niclas Alexander Göring, Chris Mingard, Yoonsoo Nam, Ard A. Louis

Introduction

Deep learning's remarkable performance is largely attributed to feature learning (FL)—the process by which a network adapts its internal representations to the data. While "infinite-width" theories like the Neural Network Gaussian Process (NNGP) offer mathematical elegance, they often fail to capture this "rich" regime where weights actively align with data patterns. Conversely, standard mean-field (MF) theories, while predicting the onset of feature learning, typically underestimate the resulting performance gains. Researchers from the University of Oxford have introduced the MF-ARD (Mean Field - Automatic Relevance Determination) model to bridge this gap, providing a tractable framework that quantitatively predicts the generalization of finite-width networks.

The MF-ARD Framework

The core challenge in modeling feature learning is capturing how networks ignore noise and focus on relevant input dimensions, a process termed Input Feature Selection (IFS). The MF-ARD model improves upon standard mean-field approaches by introducing coordinate-wise variances for weight priors. By employing a Gamma distribution prior, the model allows the network to "learn" the importance of specific input dimensions dynamically.

Theoretical hierarchy and performance comparison showing how MF-ARD matches empirical SGLD results.

This approach simplifies the complex interactions of $N$ neurons into a self-consistent problem for a single representative neuron. The authors demonstrate that this model accurately tracks the transition from "lazy" kernel-like behavior to "rich" feature-learning behavior as the training set size $P$ increases.

Mechanisms: Onset and Specialization

The research identifies feature learning as a two-stage process. First, an onset phase occurs as a symmetry-breaking phase transition where neurons begin to align with the target function. This is followed by a specialization phase, a "greedy" process where a subset of neurons becomes highly specialized, effectively "shutting out" others and creating a sparse, heavy-tailed distribution of weights.

Visualization of the feature learning phase transition and weight specialization.

Standard MF theories fail because they assume homogeneity among neurons. In contrast, MF-ARD captures the reality of SGLD (Stochastic Gradient Langevin Dynamics) training, where a few neurons do the "heavy lifting" for specific tasks.

Results: Overcoming the Curse of Dimensionality

The model was validated on "feature-sparse" tasks like $k$-sparse parity and single-index models, where only a small subset of input dimensions is relevant. A key theoretical result (Theorem 4.1) proves that MF-ARD's performance depends on the intrinsic dimension $k$ rather than the total input dimension $d$. This explains how neural networks effectively bypass the "curse of dimensionality" that plagues standard kernel methods.

Empirically, the MF-ARD model's predicted learning curves almost perfectly match those of actual SGLD-trained networks, correctly identifying the sharp boundaries where feature learning significantly reduces test error.

Limitations and Future Directions

While powerful, the current MF-ARD theory is restricted to two-layer architectures and tasks with sparse underlying structures. It also focuses on the "static" posterior distribution rather than the step-by-step training dynamics of standard SGD. Future work aims to extend this framework to deeper architectures, such as convolutional or attention-based layers, and to explore the relationship between the static theoretical predictions and the dynamic trajectories of real-world training runs.

Summary updated

Abstract

Feature learning (FL), where neural networks adapt their internal representations during training, remains poorly understood. Using methods from statistical physics, we derive a tractable, self-consistent mean-field (MF) theory for the Bayesian posterior of two-layer non-linear networks trained with stochastic gradient Langevin dynamics (SGLD). At infinite width, this theory reduces to kernel ridge regression, but at finite width it predicts a symmetry breaking phase transition where networks abruptly align with target functions. While the basic MF theory provides theoretical insight into the emergence of FL in the finite-width regime, semi-quantitatively predicting the onset of FL with noise or sample size, it substantially underestimates the improvements in generalisation after the transition. We trace this discrepancy to a key mechanism absent from the plain MF description: \textit{self-reinforcing input feature selection}. Incorporating this mechanism into the MF theory allows us to quantitatively match the learning curves of SGLD-trained networks and provides mechanistic insight into FL.
BibTeX
@misc{goring2025simple,
  author = {Niclas Alexander Göring and Chris Mingard and Yoonsoo Nam and Ard A. Louis},
  title = {{A simple mean field model of feature learning}},
  year = {2025},
  eprint = {2510.15174},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.15174},
}
Figure