Abstract: Convolutional Neural Network's (CNN's) performance disparity on clean and corrupted datasets has recently come under scrutiny. In this work, we analyse common corruptions in the frequency domain, i.e., High Frequency corruptions (HFc, e.g., noise) and Low Frequency corruptions (LFc, e.g., blur). Although a simple solution to HFc is low-pass filtering, ReLU -- a widely used Activation Function (AF), does not have any filtering mechanism. In this work, we instill low-pass filtering into the AF (LP-ReLU) to improve robustness against HFc. To deal with LFc, we complement LP-ReLU with Discrete Cosine Transform based augmentation. LP-ReLU, coupled with DCT augmentation, enables a deep network to tackle the entire spectrum of corruption. We use CIFAR-10-C and Tiny ImageNet-C for evaluation and demonstrate improvements of 5% and 7.3% in accuracy respectively, compared to the State-Of-The-Art (SOTA). We further evaluate our method's stability on a variety of perturbations in CIFAR-10-P and Tiny ImageNet-P, achieving new SOTA in these experiments as well. To further strengthen our understanding regarding CNN's lack of robustness, a decision space visualisation process is proposed and presented in this work.
Introduction
Standard Convolutional Neural Networks (CNNs) excel at classifying "clean" images but are notoriously brittle when faced with real-world corruptions like sensor noise, motion blur, or weather effects. These corruptions, often barely perceptible to humans, can cause a model's performance to drop significantly. This research identifies the ubiquitous Rectified Linear Unit (ReLU) activation function as a primary culprit. Because ReLU is unbounded for positive inputs, it allows "hyperactive" features—often triggered by high-frequency noise—to propagate through the network, causing internal representations to drift across decision boundaries.
Methods: LP-ReLU and DCT Augmentation
The proposed solution addresses corruptions by categorizing them into High-Frequency (HFc), such as Gaussian noise, and Low-Frequency (LFc), such as defocus blur.
- LP-ReLU (Low-Pass ReLU): The authors introduce a filtering mechanism directly into the activation function. Unlike standard ReLU, which passes all positive values linearly, LP-ReLU introduces thresholds. Below a threshold $A$, it behaves like ReLU. Above $A$, it attenuates the signal using a "Filtering Factor" ($\alpha$), preventing noisy, high-magnitude features from dominating.
- DCT Augmentation: To combat low-frequency corruptions like blur, the researchers employ Discrete Cosine Transform (DCT) based augmentation. This technique randomly drops frequency components during training, forcing the network to maintain robustness even when fine details (sharpness) are lost.
Analyzing Robustness via Decision Space Visualization
To validate the theoretical benefits of LP-ReLU, the authors developed a "FC2-net" visualization tool. By compressing features into a 2D space, they can map decision boundaries and observe how different components affect feature distribution.
As shown in Figure 2, standard ReLU networks produce sparse, spread-out features for a single class. In contrast, LP-ReLU enforces "feature compactness," clustering features tightly and making them harder to push across decision boundaries with noise. The combination of LP-ReLU and DCT augmentation (Figure 2d) achieves both compact features and distinctive class centers, providing the best defense against the full spectrum of corruptions.
Experimental Results
The proposed method was evaluated on several benchmarks, establishing a new state-of-the-art (SOTA) for robustness:
- Accuracy Boost: On CIFAR-10-C, the method achieved 89.2% accuracy (a 5% improvement over AugMix). Tiny ImageNet-C saw a 7.3% jump to 51.9%.
- Stability: On the "-P" (perturbation) datasets, the model demonstrated the lowest Mean Flip Probability (mFP), indicating significantly more stable predictions under sequential image alterations.
- Efficiency: Training time per epoch increased by only 10-15%, and the models often converged faster than those using standard ReLU.
Conclusion
By viewing noise as a source of neuron "hyperactivity" and addressing it at the activation level, this work provides a robust framework for image classification. LP-ReLU acts as a shield against high-frequency noise, while DCT augmentation prepares the model for low-frequency blur. Future work may extend these low-pass activation principles to other domains like Natural Language Processing or more modern architectures like Vision Transformers.
Summary updated