When training a neural network to fit a complex signal—whether it is an image, a physical simulation, or a language task—the model does not learn all parts of the signal at once. Instead, it follows a strict hierarchy: it captures broad, global structures first and only much later attempts to resolve fine-grained, high-frequency details.
This phenomenon, known as Spectral Bias or the Frequency Principle (F-Principle), suggests that neural networks act as frequency-dependent optimizers. While this bias is the secret ingredient behind the remarkable generalization of deep learning, it is also the primary bottleneck preventing models from capturing the "stiff" oscillations of physics or the sharp textures of high-resolution video.
The Intuition: Learning as a Low-Pass Filter
Imagine trying to sketch a face. You likely start with the oval of the head and the general placement of the eyes before moving on to the individual eyelashes or the texture of the skin. Neural networks trained with gradient descent behave similarly. Santosh Vempala and John Wilmes demonstrated that networks prioritize low-frequency, global patterns, establishing a fundamental learning order that transitions from simple to complex feature extraction Vempala et al..
This isn't just a preference; it is a mathematical necessity of the optimization landscape. In the early stages of training, the "error" the network sees is dominated by large-scale mismatches. Because gradient descent is a first-order optimizer, it naturally flows toward the largest reductions in error, which almost always correspond to low-frequency components. As Zhongwang Zhang notes, even when a network suffers a "loss spike" or instability, it recovers by quickly re-learning these low-frequency patterns first Zhang.
The Mathematical Root: NTK Eigenvalue Decay
To understand why this happens, researchers look to the Neural Tangent Kernel (NTK). The NTK describes how the network's outputs change in response to parameter updates. Sifan Wang and Paris Perdikaris have shown that the eigenvalues of the NTK decay rapidly as the frequency of the target function increases Wang et al..
In simple terms, the network is "numb" to high frequencies. The gradient signal for a sharp edge is orders of magnitude smaller than the signal for a smooth gradient. This creates a "hearing loss" for high-frequency data, a problem particularly acute in State Space Models Yu et al. and Physics-Informed Neural Networks (PINNs), where capturing rapid oscillations is essential for physical accuracy Kazachek et al..
Why Spectral Bias is a Double-Edged Sword
1. The Generalization Window
Spectral bias is actually why neural networks generalize instead of just memorizing noise. Since noise is typically high-frequency and irregular, the network learns the "smooth" underlying distribution long before it can fit the noise. Tony Bonnaire and Giulio Biroli argue that in diffusion models, this creates a "generalization window": the model learns global structural features early, and if training is stopped before it fits the high-frequency "memorization" patterns, it produces more creative, non-replicated samples Bonnaire et al..
2. The Blurriness Problem
Conversely, in tasks like Implicit Neural Representations (INRs)—where a network is trained to represent a single image or 3D scene—spectral bias is an enemy. It causes the model to produce blurry outputs that lack sharp edges and fine textures. Hao Zhu observes that this bias makes it difficult for coordinate-based networks to resolve intricate geometric features Zhu & Lin.
Strategies to "Hike" the Spectrum
Researchers have developed several architectural and algorithmic "hacks" to force networks to learn high frequencies faster:
- Multiplicative Interactions: Moulik Choraria and Volkan Cevher proved that Polynomial Neural Networks (PNNs)—which use Hadamard products—can drastically mitigate spectral bias, enabling much faster learning of complex details than standard additive MLPs Choraria et al..
- Frequency Separation: Architectures like SNeRV and FANeRV use wavelet transforms to explicitly split signals into high and low-frequency streams, allowing specialized modules to focus on textures and edges independently Kim et al., Yu et al..
- Adaptive Activations: Instead of standard ReLUs, functions like MetaSin Salehi or Band-Localized Activations in FLAIR Ko & Kye provide the network with the mathematical "vocabulary" needed to express high-frequency oscillations from the start.
- Coordinate Mapping: Techniques like DINER use learnable hash tables to re-arrange input coordinates, effectively "smoothing" a high-frequency signal into a low-frequency one that the network can digest more easily Zhu & Lin.
By understanding that gradient descent is a low-pass filter, we can better design architectures that either lean into this bias for better generalization or circumvent it for high-fidelity reconstruction.