In deep learning, we often assume that a model will utilize all available information to solve a task. However, a growing body of research suggests that neural networks are "lazy" optimizers. Rather than learning a comprehensive representation of the data, they often latch onto the simplest, most frequent patterns—such as textures or surface-level correlations—and ignore more complex but robust features like shape or semantics.
This phenomenon, known as Gradient Starvation or Feature Suppression, creates a fundamental bottleneck in how models generalize and transfer to new domains.
The Mechanism of Gradient Starvation
The core of the problem lies in the dynamics of gradient-based optimization. When a dataset contains a "dominant" feature—one that is easy to identify and highly predictive—the network learns it rapidly. As the model’s confidence in this feature grows, the loss function (typically cross-entropy) begins to saturate.
As Mohammad Pezeshki, Yoshua Bengio, and Aaron Courville demonstrate in Gradient Starvation: A Learning Proclivity in Neural Networks, this saturation causes the gradient signal to vanish. Once the gradient is near zero, the optimization process effectively halts, leaving secondary or "minor" features insufficiently trained. This is not a lack of model capacity, but a failure of the training procedure to explore the full parameter space, a concept Charles Sutton and Andrew McCallum described as "weight undertraining" in Reducing Weight Undertraining in Structured Discriminative Learning.
Feature Competition and Simplicity Bias
This process can be viewed as a form of Feature Competition. Jiaming Song and Stefano Ermon argue in On the Limits of Learning Representations with Label-Based Supervision that different data attributes compete for a finite budget of label-derived signal. The acquisition of an easy feature reduces the marginal signal available for others.
This "simplicity bias" is particularly visible in specific architectures and tasks:
- Transformers: JiangRui Zhao and Xiaoting Du found that easy syntactic patterns (like punctuation) can starve the gradients needed for complex logical reasoning in Syntax vs. Semantics: How Transformers Learn Deep Dependencies.
- Multi-modal Learning: In Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning?, Yu Huang and colleagues prove that modalities compete during training, often causing the network to ignore one modality entirely if the other provides a quicker path to low loss.
- CNNs: Standard convolutional networks often prioritize texture over shape because texture offers a steeper descent in the loss function, as noted by Fei Li and Rujie Liu in Learning to Find Correlated Features by Maximizing Information Flow in Convolutional Neural Networks.
Why It Matters: The Shortcut Trap
The primary danger of feature suppression is a lack of robustness. If a model only learns a "shortcut" (a spurious correlation), it will fail when that shortcut is absent. Alexander Cong Li and Ananya Kumar show in Generative Classifiers Avoid Shortcut Solutions that discriminative models are prone to this because they stop learning once they find any predictive feature.
Furthermore, this "sparsity bias" harms transfer learning. Xin Yang and Leon Bottou demonstrate in These Are Not All the Features You Are Looking For that supervised pretraining often discards features essential for downstream tasks because they weren't necessary to solve the source task.
Strategies for Promoting Feature Diversity
To counter gradient starvation, researchers have developed methods to force models to "keep looking" for features:
- Data Augmentation as a Diversifier: Muthu Chidambaram and Rong Ge prove in Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup that Mixup prevents feature suppression by forcing the network to learn multiple features for each class. Similarly, Junsoo Oh and Chulhee Yun show that Cutout and CutMix help models ignore dominant noise to capture rare features.
- Iterative Masking: The Discrimination-Aware Mechanism (DAM) by Furong Xu and Meng Wang in Discrimination-Aware Mechanism for Fine-Grained Representation Learning identifies and masks the most obvious visual cues, forcing the model to discover subtle, localized features.
- Optimization Shifts: While SGD often succumbs to simplicity bias, Bhavya Vasudeva and Mahdi Soltanolkotabi suggest that Adam is more resistant, favoring "richer" feature sets. Additionally, Sharpness-Aware Minimization (SAM) exhibits a "Minor First, Major Last" behavior, as shown by Chaewon Moon in Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization, which prioritizes weaker features early in training.
- Information Regularization: Techniques like Information Flow Maximization (IFM) force the network to maximize mutual information between layers, ensuring that a diverse set of features is preserved rather than discarded.