Abstract: Image classification has significantly improved using deep learning, mainly due to convolutional neural networks (CNNs) that learn rich feature extractors from large datasets. However, most deep learning classification methods are trained on clean images and are not robust when handling noisy ones, even if a restoration preprocessing step is applied. While novel methods address this problem, they rely on modified feature extractors and thus necessitate retraining. We instead propose a method that can be applied on a pretrained classifier. Our method exploits a fidelity map estimate that is fused into the internal representations of the feature extractor, thereby guiding the attention of the network and making it more robust to noisy data. We improve the noisy-image classification (NIC) results by significantly large margins, especially at high noise levels, and come close to the fully retrained approaches. Furthermore, as proof of concept, we show that when using our oracle fidelity map we even outperform the fully retrained methods, whether trained on noisy or restored images.
TL;DR
Researchers from EPFL have developed FG-NIC, a modular approach that allows existing, "frozen" image classifiers to handle noisy data without expensive retraining. By estimating a "fidelity map"—a pixel-wise confidence score of an image's restoration—the system guides the network's attention to trust reliable features and ignore noise-induced artifacts, significantly closing the performance gap between clean-trained and noise-robust models.
The Challenge of Noisy-Image Classification
Deep learning models for image classification are typically trained on "clean" datasets like ImageNet. However, real-world images from surveillance cameras, low-light sensors, or medical devices are often noisy. When a model trained on clean data encounters noise, its accuracy plummets.
Standard fixes usually involve either preprocessing with a denoiser—which often fails because denoisers can introduce artifacts or "smudge" important details—or retraining the entire model from scratch on noisy data. The latter is computationally expensive and discards the massive effort already put into high-quality pretrained models. The authors tackle the challenge of making pretrained, frozen classifiers robust to noise in a "plug-and-play" fashion.
Fidelity-Guided Fusion (FG-NIC)
The core innovation is Fidelity-Guided Noisy-Image Classification (FG-NIC). The process works in three main steps:
- Restoration & Fidelity Estimation: A noisy image is first passed through a restoration network (like DnCNN). Simultaneously, a small "fidelity estimator" network predicts a Fidelity Map. This map indicates, for every pixel, how much the restored image is expected to differ from the (unknown) clean original.
- Spatial Linear Fusion: This fidelity map is fused into the internal layers of the pretrained feature extractor. It uses element-wise multiplication and addition to "weight" the features, telling the network to pay more attention to high-fidelity pixels.
- Channel Fusion & Concatenation: Since CNNs also store information across channels, the authors flatten and downsample the fidelity map to guide channel-wise attention. Finally, they concatenate the fidelity information with the final feature vector before classification.
Crucially, the backbone classifier remains frozen. Only the small fusion modules and the fidelity estimator are trained, making the approach highly efficient.
Datasets and Noise Modeling
The authors evaluated their method using Caltech-256 and Caltech-101. They tested Additive White Gaussian Noise (AWGN) at various levels ($\sigma$ from 0.1 to 0.5). They also implemented spatially varying noise (1D and 2D gradients), where some parts of the image are noisier than others, to simulate real-world conditions like sensor heat or shadows.
Key Insights and Results
- Fidelity as Attention: Instead of just trying to "fix" the image, the authors use the uncertainty of the fix as a signal. This acts as a dynamic attention mechanism that helps the network ignore "hallucinated" details from the denoiser.
- Performance Gains: The improvements over the "Pretrained" baseline are substantial. For ResNet-50 on Caltech-256, the baseline accuracy on restored images was 52.33%. FG-NIC boosted this to 68.69%, coming very close to the 72.95% achieved by a model fully retrained on noise. For AlexNet, FG-NIC achieved 54.61%, actually slightly beating the fully retrained model (52.87%).
- The "Oracle" Proof of Concept: A version using a "ground truth" fidelity map (calculated using the actual clean image) actually outperformed models that were fully retrained on noisy data. This suggests that fidelity information is a powerful cue that standard training fails to capture.
Limitations and Future Work
While much cheaper than full retraining, adding the fusion modules and a fidelity estimator does increase the number of operations, though the increase is marginal (about 0.08G MACs for the FG-NIC modules vs 4.11G for the ResNet-50 backbone). The method's performance is inherently linked to the quality of the initial restoration network. Future work could explore more complex degradations like motion blur, haze, or compression artifacts beyond Gaussian noise.
Summary updated