Abstract
Despite their ability to represent highly expressive functions, deep learning models seem to find simple solutions that generalize surprisingly well. Spectral bias—the tendency of neural networks to prioritize learning low-frequency functions—is one possible explanation for this phenomenon. This work proposes methodologies for measuring spectral bias in modern image classification networks on CIFAR-10 and ImageNet. We find that these networks indeed exhibit spectral bias and that interventions improving test accuracy tend to produce learned functions with higher frequencies overall but lower frequencies in the vicinity of examples from each class. This trend holds across variation in training time, model architecture, data augmentation, and self-distillation.
Introduction: The Mystery of Generalization
A central question in deep learning is why "overparameterized" models—those with far more parameters than training examples—generalize to new data instead of simply memorizing the training set. A leading theory is spectral bias: the idea that the optimization process (like SGD) inherently prioritizes simple, smooth, low-frequency functions. While theoretical evidence exists, measuring this in large-scale networks like ResNets or Vision Transformers has been challenging. This study bridges that gap by providing tools to measure spectral bias in practical settings.
Measuring Function Frequency
The authors propose two primary methodologies to quantify the "frequency" of a learned neural network function:
- Multi-class Label Smoothing (Noise Fitting): By adding sinusoidal noise of a specific frequency to training labels and monitoring how much the model "fits" this noise versus the true labels, researchers can determine which frequencies the model finds easiest to learn.
- Linear Interpolation Paths: This direct measurement involves creating a linear path of "blended" images between two samples (from the same or different classes). A Discrete Fourier Transform (DFT) is then performed on the model's predictions along that path. High-frequency functions show rapid prediction changes, while low-frequency functions change gradually.
Key Insights: The "Good" Frequency Gap
The study evaluates modern architectures (Wide-ResNet, Shake-Shake, ViT, etc.) on CIFAR-10 and ImageNet, revealing several critical insights:
Within-Class vs. Between-Class Smoothness
The most successful models are not uniformly "smooth." Instead, they exhibit a massive disparity: they are very low-frequency (smooth) within a class but high-frequency (sharp) at the boundaries between classes. This "frequency gap" validates the intuition that a good classifier should have sharp decision boundaries while remaining stable for variations of the same object.
Accuracy and Frequency Link
Interventions that improve test accuracy—such as longer training, more data, or advanced data augmentation—consistently increase this frequency gap. For instance, self-distillation acts like a low-pass filter, making the "student" model lower-frequency within-class than its "teacher," which correlates with better generalization.
Sensitivity to Natural Image Statistics
Neural networks appear "tuned" to the statistics of the real world. Since natural images are dominated by low frequencies, the spectral bias of these models aligns with the inherent structure of the data they process. On ImageNet, models learn higher-frequency functions for more "diverse" classes (e.g., "dog") compared to highly consistent classes (e.g., "poodle").
Conclusion and Future Directions
This work confirms that spectral bias is a real phenomenon in modern deep learning. The discovery of the "frequency gap" provides a new lens through which to evaluate model performance. Future work may involve interpolating in the latent space of generative models to avoid "off-manifold" artifacts in pixel-space interpolation. Ultimately, these frequency metrics could be used as explicit regularizers to force models toward smoother within-class representations, potentially leading to more robust and generalizable AI.
Summary updated