How a Single Gradient Step Triggers Feature Learning

39 concepts27 papers1 proposal

The Lazy versus Rich Dichotomy

In deep learning theory, neural networks are often analyzed through two contrasting lenses: the lazy training regime and the rich feature-learning regime. In the lazy regime, an overparameterized neural network behaves essentially like a linear model operating over a fixed kernel—specifically, the Neural Tangent Kernel (NTK). Here, the hidden layer weights barely move from their random initialization. As shown by Mario Geiger et al. in Disentangling feature and lazy learning in deep neural networks: an empirical study, whether a network falls into the lazy or rich regime is governed by a scaling parameter of the final layer relative to the network's width. When this scaling is high, the network remains trapped in a static, rotation-invariant kernel space.

However, lazy models suffer from severe performance limitations when dealing with complex, high-dimensional target functions. Behrooz Ghorbani et al. demonstrated in Limitations of Lazy Training of Two-layers Neural Network. that fully trained networks significantly outperform lazy models because they actively rotate their neuron weights to align with the principal eigenvectors of the target function's underlying structure. This active adaptation defines the rich feature-learning regime. As reviewed by Cong Fang et al. in Mathematical Models of Overparameterized Neural Networks, understanding how optimization navigates the boundary between these two regimes is central to modern deep learning theory.

The Single-Step Proxy

To isolate exactly how a network breaks out of the lazy regime, researchers use an elegant theoretical proxy: updating the first-layer weights with a single gradient descent step, followed by optimizing the second layer via a linear readout or ridge regression.

This two-stage optimization framework—analyzed by Jimmy Ba et al. in Learning in the Presence of Low-dimensional Structure: A Spiked Random Matrix Perspective and Behrad Moniri et al. in A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks—allows us to see the immediate impact of gradient information.

Why does just one step matter? At initialization, the hidden neurons point in completely random directions. When a batch of data passes through the network, the resulting gradient update isn't purely random noise; it contains a structured signal derived from the target function. By taking a single gradient step with a sufficiently aggressive learning rate, the network applies a low-rank transformation to the weight matrix. This transformation rotates the hidden neurons away from their random initialization and aligns them with the principal non-linearities or "spikes" in the data's underlying structure, a phenomenon mathematically detailed by Samet Demir et al. in Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure.

The Mathematics of the "Spike"

To build intuition for why this works, consider the spectrum (the distribution of eigenvalues) of the network's weight matrix. At initialization, the spectrum resembles a uniform bulk of random noise, meaning the network treats all input directions isotropically.

As Jimmy Ba et al. demonstrate in High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation, the initial gradient update acts as a low-rank matrix that pulls the hidden layer weights toward the target task's primary directions. In the weight spectrum, this causes specific "outlier eigenvalues"—or spikes—to emerge from the random bulk.

Zhichao Wang shows in Spectral Evolution and Invariance in Linear-width Neural Networks that increasing the learning rate past a critical threshold triggers a sharp phase transition where these outlier spikes suddenly appear. This spectral shift means the weights have developed a structural alignment with the target function. As proven by Hugo Cui et al. in Asymptotics of feature learning in two-layer networks after one gradient-step, this single-step low-rank alignment is mathematically sufficient to break out of the lazy kernel regime, allowing the network to capture complex non-linear patterns that fixed-kernel methods require exponential data to resolve.

Amplifying Weak Signals and Breaking Hard Problems

Why is this transition so powerful? Certain problems, like sparse parities or high-dimensional hidden signals, are fundamentally invisible to static linear embeddings or random feature models. Linear models require exponential resources or sample sizes to solve them because they cannot adapt to the data's geometry.

Gradient descent overcomes this through feature amplification. As explained by Spencer Frei et al. in Random Feature Amplification: Feature Learning and Generalization in Neural Networks, the initial gradient step takes the tiny, weak correlations present between random neurons and data clusters at initialization and amplifies them into strong, significant features.

This mechanism provides an exponential separation between neural networks and kernel methods. Amit Daniely et al. demonstrated in Learning Parities with Neural Networks that initial gradient steps fundamentally transform random weights into structured features capable of solving parity logic in polynomial time. By aligning the weights early on, the network extracts task-relevant representations that act as a foundation for all subsequent training, a concept formalized by Zhenmei Shi et al. in Provable Guarantees for Neural Networks via Gradient Feature Learning.

The Suddenness of Feature Learning: Grokking and Stability

This transition from lazy memorization to rich feature learning isn't always a gradual, linear process; it frequently manifests as a sudden, sharp phase transition.

For instance, the phenomenon of grokking—where a model suddenly generalizes long after overfitting—is fundamentally a delayed transition from a lazy kernel regime to a rich feature-learning regime. Mohamad Amin Mohamadi et al. show in Why Do You Grok? A Theoretical Analysis on Grokking Modular Addition that the model initially relies on data-intensive memorization before shifting into a rich regime characterized by low-norm, generalizing features. This structural shift from a memorization circuit to an algorithmic circuit is typically driven by optimization dynamics and regularization pressures, as explored by Bilal Chughtai et al. in A Toy Model of Universality: Reverse Engineering how Networks Learn Group Operations and quantified by Yoonsoo Nam et al. in Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement.

What triggers this sudden shift? It can be explicit or implicit regularization that aligns with the target solution's structure, such as sparsity or low-rankness, as shown by Pascal Junior Tikeng Notsawo et al. in Grokking Beyond the Euclidean Norm of Model Parameters. Alternatively, it can be driven by optimization noise after reaching zero training loss, which allows the model to escape the kernel regime, as detailed by Zhiyuan Li et al. in What Happens after SGD Reaches Zero Loss? --A Mathematical Framework. Furthermore, this transition can be accelerated by techniques like dropout and weight decay, even in large language models learning fundamental capabilities like in-context copying, as demonstrated by Ang Lv et al. in Language Models "Grok" to Copy.

Crucially, this jump often requires operating at the "edge of stability." Kwangjun Ahn et al. demonstrated in Learning threshold neurons via edge of stability that a network's ability to acquire specific feature representations depends on the learning rate exceeding a critical threshold. Below this value, the model remains trapped in a poorly generalizing state; above it, the optimization dynamics destabilize just enough to force the weights to snap into alignment with essential features.

This sudden 'click' moment in representation geometry, also observed in recurrent architectures by Aldo Battista et al. (Neural representational geometries reflect behavioral differences in monkeys and recurrent neural networks) and Hsiang Hsu et al. (Deep Orthogonal Representations: Fundamental Properties and Applications), marks the definitive boundary where a network stops acting like a static kernel and begins acting like a true feature learner.

Key Foundational Work

Deep Orthogonal Representations: Fundamental Properties and Applications. (2018) — contributed 1 concept elements to this direction.
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019) — contributed 1 concept elements to this direction.
Limitations of Lazy Training of Two-layers Neural Network. (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_single_step_feature_learning_bc73bc28,
  title = {single-step feature learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-step-feature-learning/rdr_bc73bc2825fb48e7b1bd774f499947ae},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_single_step_feature_learning_bc73bc28_v1,
  title = {single-step feature learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-step-feature-learning/rdr_bc73bc2825fb48e7b1bd774f499947ae/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake