Abstract
Model robustness is vital for the reliable deployment of machine learning models in real-world applications. Recent studies have shown that data augmentation can result in model over-relying on features in the low-frequency domain, sacrificing performance against low-frequency corruptions, highlighting a connection between frequency and robustness. Here, we take one step further to more directly study the frequency bias of a model through the lens of its Jacobians and its implication to model robustness. To achieve this, we propose Jacobian frequency regularization for models' Jacobians to have a larger ratio of low-frequency components. Through experiments on four image datasets, we show that biasing classifiers towards low (high)-frequency components can bring performance gain against high (low)-frequency corruption and adversarial perturbation, albeit with a tradeoff in performance for low (high)-frequency corruption. Our approach elucidates a more direct connection between the frequency bias and robustness of deep learning models.
Introduction
Deep learning models are notoriously brittle. A model that perfectly identifies a dog in a clear photo might fail if the image is slightly blurry, foggy, or subjected to "adversarial perturbations"—tiny, invisible-to-humans pixel changes designed to fool the AI. Previous research suggested a link between a model's "frequency bias" and its robustness. For example, natural images are dominated by low-frequency components (broad shapes and smooth colors), while the noise used in adversarial attacks is often high-frequency (sharp, jagged changes). Standard neural networks often "over-rely" on high-frequency features that humans do not prioritize. This paper addresses the lack of a direct way to measure and manipulate this frequency bias during training to see exactly how it impacts a model's ability to handle different types of noise.
Methodology: Jacobian Frequency Regularization (JaFR)
The authors introduce a novel training technique called Jacobian Frequency Regularization (JaFR) to explicitly control whether a neural network relies on low-frequency or high-frequency image features.
The process follows these key steps:
- The Jacobian ($J$): Instead of looking at the image itself, the method analyzes the model's Jacobian—a map showing which pixels most influence the model's final prediction.
- Fourier Transform: A Discrete Fourier Transform (DFT) is applied to this Jacobian to reveal its "Fourier profile," indicating whether the model focuses on low-frequency shapes or high-frequency textures.
- Frequency Bias Term ($B_{low}$): A differentiable scalar value is proposed to quantify the ratio of low-to-high frequency components in the Jacobian.
- Regularization: During training, a new loss term ($\mathcal{L}_{freq}$) is added to encourage the model to maximize (or minimize) this $B_{low}$ value. By adjusting a single hyperparameter ($\lambda_{freq}$), researchers can "steer" the model to be more low-frequency or high-frequency biased.
Experimental Results
The team evaluated JaFR on standard benchmarks including SVHN, CIFAR-10, CIFAR-100, and TinyImageNet, along with their "corrupted" counterparts (CIFAR-10-C/100-C).
Visual Saliency and Frequency Mismatch
Standard-trained models have Jacobians dominated by high-frequency noise, whereas the actual images they are classifying are dominated by low-frequency signals. JaFR helps bridge this gap. When JaFR is applied, the model's Jacobians look "cleaner" and more like the actual objects in the image, suggesting the model is learning more "human-like" features.
Robustness and Efficiency
- Corruption Robustness: On CIFAR-10-C, the JaFR-only model achieved a mean Corruption Error (mCE) of 95.84, outperforming standard models (100.00). It was particularly effective against high-frequency noise like "Shot Noise" and "Impulse Noise."
- Adversarial Defense: Combining JaFR with weak adversarial training (FGSM) boosted adversarial accuracy on CIFAR-100 from 0.04% to 23.42%, nearly matching the performance of the much slower "gold standard" PGD AT (25.24%) while being significantly faster to train.
Discussion and Limitations
There is a fundamental frequency-dependent tradeoff: biasing a model toward low frequencies makes it much better at resisting high-frequency noise and adversarial attacks, but it makes the model worse at handling low-frequency corruptions like fog or changes in contrast. Because JaFR forces the model to ignore certain frequency bands, it inherently becomes vulnerable to corruptions that exist in those bands. Future work could explore dynamic frequency-focused approaches where models adaptively shift their frequency bias depending on the input.
Summary updated