Gradient Starvation and Feature Suppression

This direction is split into other directions: gradient starvation. Its last version stays readable and citable here.

Version 1 of 135 papers49 conceptsfrozen 2026-10-04

In deep learning, we often assume that a model will utilize all available information to solve a task. However, a growing body of research suggests that neural networks are "lazy" optimizers. Rather than learning a comprehensive representation of the data, they often latch onto the simplest, most frequent patterns—such as textures or surface-level correlations—and ignore more complex but robust features like shape or semantics.

This phenomenon, known as Gradient Starvation or Feature Suppression, creates a fundamental bottleneck in how models generalize and transfer to new domains.

The Mechanism of Gradient Starvation

The core of the problem lies in the dynamics of gradient-based optimization. When a dataset contains a "dominant" feature—one that is easy to identify and highly predictive—the network learns it rapidly. As the model’s confidence in this feature grows, the loss function (typically cross-entropy) begins to saturate.

As Mohammad Pezeshki, Yoshua Bengio, and Aaron Courville demonstrate in Gradient Starvation: A Learning Proclivity in Neural Networks, this saturation causes the gradient signal to vanish. Once the gradient is near zero, the optimization process effectively halts, leaving secondary or "minor" features insufficiently trained. This is not a lack of model capacity, but a failure of the training procedure to explore the full parameter space, a concept Charles Sutton and Andrew McCallum described as "weight undertraining" in Reducing Weight Undertraining in Structured Discriminative Learning.

Feature Competition and Simplicity Bias

This process can be viewed as a form of Feature Competition. Jiaming Song and Stefano Ermon argue in On the Limits of Learning Representations with Label-Based Supervision that different data attributes compete for a finite budget of label-derived signal. The acquisition of an easy feature reduces the marginal signal available for others.

This "simplicity bias" is particularly visible in specific architectures and tasks:

Why It Matters: The Shortcut Trap

The primary danger of feature suppression is a lack of robustness. If a model only learns a "shortcut" (a spurious correlation), it will fail when that shortcut is absent. Alexander Cong Li and Ananya Kumar show in Generative Classifiers Avoid Shortcut Solutions that discriminative models are prone to this because they stop learning once they find any predictive feature.

Furthermore, this "sparsity bias" harms transfer learning. Xin Yang and Leon Bottou demonstrate in These Are Not All the Features You Are Looking For that supervised pretraining often discards features essential for downstream tasks because they weren't necessary to solve the source task.

Strategies for Promoting Feature Diversity

To counter gradient starvation, researchers have developed methods to force models to "keep looking" for features:

  1. Data Augmentation as a Diversifier: Muthu Chidambaram and Rong Ge prove in Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup that Mixup prevents feature suppression by forcing the network to learn multiple features for each class. Similarly, Junsoo Oh and Chulhee Yun show that Cutout and CutMix help models ignore dominant noise to capture rare features.
  2. Iterative Masking: The Discrimination-Aware Mechanism (DAM) by Furong Xu and Meng Wang in Discrimination-Aware Mechanism for Fine-Grained Representation Learning identifies and masks the most obvious visual cues, forcing the model to discover subtle, localized features.
  3. Optimization Shifts: While SGD often succumbs to simplicity bias, Bhavya Vasudeva and Mahdi Soltanolkotabi suggest that Adam is more resistant, favoring "richer" feature sets. Additionally, Sharpness-Aware Minimization (SAM) exhibits a "Minor First, Major Last" behavior, as shown by Chaewon Moon in Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization, which prioritizes weaker features early in training.
  4. Information Regularization: Techniques like Information Flow Maximization (IFM) force the network to maximize mutual information between layers, ensuring that a diverse set of features is preserved rather than discarded.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gradient_starvation_and_feature_suppress_5a57ed27,
  title = {Gradient Starvation and Feature Suppression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-starvation-and-feature-suppression/rdr_5a57ed2759974d609b7edc6f9ca8adee},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_gradient_starvation_and_feature_suppress_5a57ed27_v1,
  title = {Gradient Starvation and Feature Suppression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-starvation-and-feature-suppression/rdr_5a57ed2759974d609b7edc6f9ca8adee/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}