Spectral Bias in Practice: The Role of Function Frequency in Generalization

Sara Fridovich-Keil, Raphael Gontijo Lopes, Rebecca Roelofs

Abstract

Despite their ability to represent highly expressive functions, deep learning models seem to find simple solutions that generalize surprisingly well. Spectral bias—the tendency of neural networks to prioritize learning low-frequency functions—is one possible explanation for this phenomenon. This work proposes methodologies for measuring spectral bias in modern image classification networks on CIFAR-10 and ImageNet. We find that these networks indeed exhibit spectral bias and that interventions improving test accuracy tend to produce learned functions with higher frequencies overall but lower frequencies in the vicinity of examples from each class. This trend holds across variation in training time, model architecture, data augmentation, and self-distillation.

Introduction: The Mystery of Generalization

A central question in deep learning is why "overparameterized" models—those with far more parameters than training examples—generalize to new data instead of simply memorizing the training set. A leading theory is spectral bias: the idea that the optimization process (like SGD) inherently prioritizes simple, smooth, low-frequency functions. While theoretical evidence exists, measuring this in large-scale networks like ResNets or Vision Transformers has been challenging. This study bridges that gap by providing tools to measure spectral bias in practical settings.

Measuring Function Frequency

The authors propose two primary methodologies to quantify the "frequency" of a learned neural network function:

  1. Multi-class Label Smoothing (Noise Fitting): By adding sinusoidal noise of a specific frequency to training labels and monitoring how much the model "fits" this noise versus the true labels, researchers can determine which frequencies the model finds easiest to learn.
  2. Linear Interpolation Paths: This direct measurement involves creating a linear path of "blended" images between two samples (from the same or different classes). A Discrete Fourier Transform (DFT) is then performed on the model's predictions along that path. High-frequency functions show rapid prediction changes, while low-frequency functions change gradually.
Methodology for measuring function frequency using linear interpolation paths and Discrete Fourier Transform (DFT).

Key Insights: The "Good" Frequency Gap

The study evaluates modern architectures (Wide-ResNet, Shake-Shake, ViT, etc.) on CIFAR-10 and ImageNet, revealing several critical insights:

Within-Class vs. Between-Class Smoothness

The most successful models are not uniformly "smooth." Instead, they exhibit a massive disparity: they are very low-frequency (smooth) within a class but high-frequency (sharp) at the boundaries between classes. This "frequency gap" validates the intuition that a good classifier should have sharp decision boundaries while remaining stable for variations of the same object.

High frequency fraction of the learned function across different architectures, showing the gap between within-class and between-class paths.

Interventions that improve test accuracy—such as longer training, more data, or advanced data augmentation—consistently increase this frequency gap. For instance, self-distillation acts like a low-pass filter, making the "student" model lower-frequency within-class than its "teacher," which correlates with better generalization.

Sensitivity to Natural Image Statistics

Neural networks appear "tuned" to the statistics of the real world. Since natural images are dominated by low frequencies, the spectral bias of these models aligns with the inherent structure of the data they process. On ImageNet, models learn higher-frequency functions for more "diverse" classes (e.g., "dog") compared to highly consistent classes (e.g., "poodle").

Conclusion and Future Directions

This work confirms that spectral bias is a real phenomenon in modern deep learning. The discovery of the "frequency gap" provides a new lens through which to evaluate model performance. Future work may involve interpolating in the latent space of generative models to avoid "off-manifold" artifacts in pixel-space interpolation. Ultimately, these frequency metrics could be used as explicit regularizers to force models toward smoother within-class representations, potentially leading to more robust and generalizable AI.

Summary updated

Abstract

Despite their ability to represent highly expressive functions, deep learning models seem to find simple solutions that generalize surprisingly well. Spectral bias -- the tendency of neural networks to prioritize learning low frequency functions -- is one possible explanation for this phenomenon, but so far spectral bias has primarily been observed in theoretical models and simplified experiments. In this work, we propose methodologies for measuring spectral bias in modern image classification networks on CIFAR-10 and ImageNet. We find that these networks indeed exhibit spectral bias, and that interventions that improve test accuracy on CIFAR-10 tend to produce learned functions that have higher frequencies overall but lower frequencies in the vicinity of examples from each class. This trend holds across variation in training time, model architecture, number of training examples, data augmentation, and self-distillation. We also explore the connections between function frequency and image frequency and find that spectral bias is sensitive to the low frequencies prevalent in natural images. On ImageNet, we find that learned function frequency also varies with internal class diversity, with higher frequencies on more diverse classes. Our work enables measuring and ultimately influencing the spectral behavior of neural networks used for image classification, and is a step towards understanding why deep models generalize well.
BibTeX
@inproceedings{fridovichkeil2022spectral,
  author = {Sara Fridovich-Keil and Raphael Gontijo Lopes and Rebecca Roelofs},
  title = {{Spectral Bias in Practice: The Role of Function Frequency in Generalization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=s5rF3EBpJP},
}
Figure