Abstract
While convolutional neural networks (CNNs) are widely recognized for their ability to extract complex visual features, the specific mechanisms by which they prioritize information remain a subject of active research. This paper investigates the relationship between frequency sensitivity, object scale, and classification performance. We demonstrate that CNNs do not merely possess a general "low-frequency bias" but instead exhibit class-specific and scale-dependent frequency preferences. To validate the necessity of multi-scale frequency extraction, we introduce the Gaussian Scale-Space Derivative Network (GSSDNet), an architecture that replaces learned spatial filters with classical Gaussian derivatives. Our results show that GSSDNet achieves performance parity with standard "black box" networks, suggesting that multi-scale derivative extraction is a fundamental requirement for computer vision.
Introduction
Despite the ubiquity of CNNs, understanding how these models prioritize visual information is critical for improving robustness and efficiency. Previous studies have suggested that CNNs favor low-frequency information—the "blurry" global structure of an image—over high-frequency details like sharp edges. This bias is often cited as a reason for the vulnerability of models to high-frequency adversarial perturbations. However, the interaction between an object's scale (its size within the frame) and its frequency (the rate of change in its features) has not been fully explored. This research aims to map these relationships and determine if feature extraction can be explicitly modeled using classical mathematical principles.
Frequency Sensitivity Analysis
To understand how networks utilize different frequency bands, we conducted a systematic sensitivity analysis across several popular architectures, including ResNet, MobileNetV3, Vision Transformers (ViT), and Swin Transformers. By filtering ImageNet images through Low-Pass Filters (LPF) and High-Pass Filters (HPF), we isolated specific frequency components to observe their impact on model accuracy.
Class-Specific Fingerprints
Our analysis revealed that frequency requirements are not uniform across all categories. Different object classes exhibit unique frequency "fingerprints." For instance, certain classes (e.g., "redbone" dogs) are identifiable via a narrow band of low frequencies, whereas other categories require a broader spectrum to be distinguished from visually similar classes.
The Scale-Frequency Shift
We observed a significant shift in frequency preference based on object scale. When an object's scale increases (i.e., it occupies more of the frame), the network's preferred frequency band shifts toward the lower end of the spectrum. This confirms that scale and frequency are intrinsically linked in the context of feature extraction; the "importance" of a frequency band is relative to the size of the features it represents.
GSSDNet: Gaussian Scale-Space Derivative Network
Inspired by classical "Scale-Space Theory," we developed GSSDNet to test whether the spatial filters learned by standard CNNs could be replaced by fixed mathematical operations. The GSSDNet building block utilizes:
- Gaussian Blurring: To establish a scale space, representing the image at varying levels of smoothness.
- Partial Derivatives: To serve as local feature extractors, identifying edges and gradients at each scale.
- Pointwise Convolutions (1x1): To learn how to combine these mathematically derived features into high-level representations.
By constraining the spatial extraction to Gaussian derivatives, we move away from arbitrary weight learning toward a more interpretable, structured approach to vision.
Experimental Results
We evaluated GSSDNet against standard depthwise separable CNNs (VanillaNets) across three benchmarks: MNIST, CIFAR-100, and ImageNet-1K.
- Small to Mid-Scale Datasets: On MNIST and CIFAR-100, GSSDNet outperformed the vanilla baselines. Specifically, on CIFAR-100, GSSDNet achieved 77.1% accuracy compared to the baseline's 76.1%.
- Large-Scale Classification: On ImageNet-1K, GSSDNet reached 71.2% Top-1 accuracy, performing nearly as well as the VanillaNet baseline of 71.8%.
- Frequency Redundancy: Our experiments showed that while pre-trained models rely heavily on low-to-medium frequencies, they can be retrained to utilize high-frequency data. For example, retraining on filtered data saw accuracy jump from 53.1% to 70.6% when low frequencies were removed, indicating that standard training regimes often leave the high-frequency spectrum underutilized.
Discussion and Conclusion
The performance of GSSDNet suggests that standard CNNs are likely learning to approximate classical Gaussian derivatives during training. By "hard-coding" these mathematical principles into the architecture, we achieve comparable results with greater interpretability.
However, a small performance gap remains on complex datasets like ImageNet, suggesting that standard convolutions may capture certain non-Gaussian features that our current derivative-based approach misses. Future research will focus on the computational efficiency of these structures and their potential to improve model robustness against adversarial attacks and common image corruptions.
Summary updated