Deep neural networks are often described as "lazy" learners. Despite their immense capacity to represent complex functions, they exhibit a consistent preference for the simplest possible patterns that can satisfy a training objective. This phenomenon, known as simplicity bias, explains why models frequently rely on superficial "shortcuts"—such as background textures or local statistical correlations—while ignoring the robust, causal structures that humans use to understand the world.
The Path of Least Resistance
The core of simplicity bias lies in the dynamics of gradient-based optimization. During the early stages of training, neural networks do not explore all possible features equally. Instead, they prioritize features that offer the fastest reduction in training loss. Pezeshki et al. formalize this as "Gradient Starvation": once a model identifies a simple, high-magnitude feature that distinguishes classes, the resulting gradients "starve" other potential features, preventing the network from learning more complex but potentially more reliable signals.
This preference is mathematically rooted in the architecture and optimization process itself. Frei et al. demonstrate that for two-layer Leaky ReLU networks, gradient descent naturally biases the model toward low-rank, linear-like classifiers, even when the data requires non-linear boundaries. Similarly, Abbe et al. show that networks tend to adopt low-degree polynomial solutions for logical tasks, failing to generalize to the higher-degree complexities required for true algorithmic reasoning.
Shortcut Learning and the Generalization Gap
While simplicity bias allows models to converge quickly, it creates a significant vulnerability: shortcut learning. Because simple features like color or texture are often computationally easier to identify than complex object structures, models converge on these superficial attributes first (Manduchi et al.).
This behavior is particularly evident in Transformers and Large Language Models. Kawata et al. argue that Transformers naturally gravitate toward positional shortcuts—simple mathematical mappings of token distances—rather than the content-based retrieval necessary for induction. In arithmetic, Deng et al. suggest that LLMs act as "greedy symbolic learners," mastering a hierarchy of simple symbol-to-symbol shortcuts rather than the underlying mathematical algorithms.
The consequence is a failure in out-of-distribution (OOD) generalization. As Tsoy et al. observe, simplicity bias often intensifies as training progresses toward zero loss. The model increasingly relies on these easy shortcuts, causing OOD performance to peak mid-training and then decline as the model over-specializes on training-set artifacts.
Strategies to Break the Bias
To build more robust models, researchers have developed techniques to force neural networks away from the path of least resistance:
- Feature Diversification: Methods like Midpoint Mixup (Chidambaram et al.) and Spectral Decoupling (Pezeshki et al.) mathematically penalize the dominance of a few simple features, forcing the network to learn a more diverse set of predictive patterns.
- Diversify and Disambiguate (DivDis): Lee et al. propose a two-stage framework that first encourages a model to find multiple competing explanations for the data and then uses a small amount of targeted information to select the one that generalizes best.
- Causal Intervention: By using causal matching (Mahajan et al.) or information minimization (Patil et al.), researchers can identify and subtract spurious biases from feature representations, ensuring the model relies on stable, invariant "objects" rather than environmental noise.
- Optimization Scheduling: Interestingly, Chen et al. found that briefly suppressing the learning of simple syntactic structures early in training can lead to more powerful final models, suggesting that the timing of feature acquisition is critical.
Ultimately, simplicity bias is a double-edged sword. It provides the inductive bias necessary for models to learn efficiently from finite data, but without careful intervention, it traps them in a world of superficial correlations. Understanding and mitigating this bias is central to moving from models that merely "fit" data to models that truly reason.