Introduction
Deep learning's remarkable performance is largely attributed to feature learning (FL)—the process by which a network adapts its internal representations to the data. While "infinite-width" theories like the Neural Network Gaussian Process (NNGP) offer mathematical elegance, they often fail to capture this "rich" regime where weights actively align with data patterns. Conversely, standard mean-field (MF) theories, while predicting the onset of feature learning, typically underestimate the resulting performance gains. Researchers from the University of Oxford have introduced the MF-ARD (Mean Field - Automatic Relevance Determination) model to bridge this gap, providing a tractable framework that quantitatively predicts the generalization of finite-width networks.
The MF-ARD Framework
The core challenge in modeling feature learning is capturing how networks ignore noise and focus on relevant input dimensions, a process termed Input Feature Selection (IFS). The MF-ARD model improves upon standard mean-field approaches by introducing coordinate-wise variances for weight priors. By employing a Gamma distribution prior, the model allows the network to "learn" the importance of specific input dimensions dynamically.
This approach simplifies the complex interactions of $N$ neurons into a self-consistent problem for a single representative neuron. The authors demonstrate that this model accurately tracks the transition from "lazy" kernel-like behavior to "rich" feature-learning behavior as the training set size $P$ increases.
Mechanisms: Onset and Specialization
The research identifies feature learning as a two-stage process. First, an onset phase occurs as a symmetry-breaking phase transition where neurons begin to align with the target function. This is followed by a specialization phase, a "greedy" process where a subset of neurons becomes highly specialized, effectively "shutting out" others and creating a sparse, heavy-tailed distribution of weights.
Standard MF theories fail because they assume homogeneity among neurons. In contrast, MF-ARD captures the reality of SGLD (Stochastic Gradient Langevin Dynamics) training, where a few neurons do the "heavy lifting" for specific tasks.
Results: Overcoming the Curse of Dimensionality
The model was validated on "feature-sparse" tasks like $k$-sparse parity and single-index models, where only a small subset of input dimensions is relevant. A key theoretical result (Theorem 4.1) proves that MF-ARD's performance depends on the intrinsic dimension $k$ rather than the total input dimension $d$. This explains how neural networks effectively bypass the "curse of dimensionality" that plagues standard kernel methods.
Empirically, the MF-ARD model's predicted learning curves almost perfectly match those of actual SGLD-trained networks, correctly identifying the sharp boundaries where feature learning significantly reduces test error.
Limitations and Future Directions
While powerful, the current MF-ARD theory is restricted to two-layer architectures and tasks with sparse underlying structures. It also focuses on the "static" posterior distribution rather than the step-by-step training dynamics of standard SGD. Future work aims to extend this framework to deeper architectures, such as convolutional or attention-based layers, and to explore the relationship between the static theoretical predictions and the dynamic trajectories of real-world training runs.
Summary updated