For years, the theoretical understanding of neural networks was dominated by the "lazy training" regime. In this view, wide neural networks behave like fixed kernels—specifically the Neural Tangent Kernel (NTK)—where weights barely move from their initialization. While mathematically elegant, this perspective fails to capture the most important property of deep learning: feature learning, the process by which a network adapts its internal representation to the specific structure of a task.
A growing body of research, led by researchers like Jimmy Ba, Murat A. Erdogdu, and Denny Wu, has recently demonstrated that we don't need thousands of iterations to see this adaptation. Instead, a single, well-scaled gradient descent step is sufficient to "break" the lazy regime and align a network’s neurons with the underlying signal of the data.
The Two-Stage Proxy for Feature Learning
To study this phenomenon rigorously, researchers often use a simplified two-stage training procedure. First, they apply a single gradient descent update to the first-layer weights of a two-layer neural network. Second, they freeze those weights and optimize the output layer using ridge regression.
As explored in A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks by Behrad Moniri and colleagues, this setup serves as a "minimal model" for feature learning. It isolates the impact of the initial weight update, allowing theorists to prove exactly how the representation changes before the final task-specific optimization occurs.
Alignment via Spiked Random Matrices
The intuition behind why one step works lies in alignment. At initialization, the weights of a neural network are essentially random noise. A gradient step acts as a low-rank transformation—specifically a rank-1 or low-rank "spike"—that pulls these random weights toward the primary directions of the target function.
In Learning in the Presence of Low-dimensional Structure: A Spiked Random Matrix Perspective, Jimmy Ba and his co-authors show that this update creates a "spiked" structure in the weight matrix. While the bulk of the weights remains random, a small number of directions now point directly at the "signal" hidden in the high-dimensional noise. This alignment allows the network to capture non-linear patterns that fixed-kernel methods, like Kernel Ridge Regression, would require significantly more data to see.
Breaking the Lazy Regime
The transition from a fixed kernel to a learned representation is not subtle. Hugo Cui, Luca Pesce, and Yatin Dandi demonstrate in Asymptotics of feature learning in two-layer networks after one gradient-step that a large learning rate is the key catalyst.
When the learning rate is sufficiently high, the first gradient step moves the weights far enough from their initialization to fundamentally change the network's geometry. This "breakout" enables the model to capture complex, task-specific features that are mathematically inaccessible to the NTK. Essentially, the single step transforms the network from a generic function approximator into one tailored to the specific non-linearities of the dataset.
Polynomial Equivalence and Complexity
What exactly does the network learn in that one step? Research by Samet Demir and Zafer Dogan in Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure suggests a "polynomial equivalence."
They prove that a network updated with one gradient step behaves like a structured polynomial predictor. The "degree" of the features the network can learn is determined by the interplay between the data's variance and the learning rate. By scaling the learning rate, the network can sequentially capture higher-order polynomial components of the target function, effectively "tuning" the complexity of its learned features in a single update.
Why It Matters
This line of research provides a theoretical bridge between the simplicity of linear models and the black-box complexity of deep learning. By showing that High-dimensional Asymptotics of Feature Learning can be characterized by a single step, researchers like Taiji Suzuki and Zhichao Wang are uncovering the fundamental efficiency of gradient-based optimization. It suggests that the "magic" of deep learning—its ability to find needles of signal in haystacks of noise—begins almost the moment training starts.