Fidelity Estimation Improves Noisy-Image Classification With Pretrained Networks

Xiaoyu Lin, Deblina Bhattacharjee, Majed El Helou, Sabine Süsstrunk

Abstract

Image classification has significantly improved using deep learning. This is mainly due to convolutional neural networks (CNNs) that are capable of learning rich feature extractors from large datasets. However, most deep learning classification methods are trained on clean images and are not robust when handling noisy ones, even if a restoration preprocessing step is applied. While novel methods address this problem, they rely on modified feature extractors and thus necessitate retraining. We instead propose a method that can be applied on a $pretrained$ classifier. Our method exploits a fidelity map estimate that is fused into the internal representations of the feature extractor, thereby guiding the attention of the network and making it more robust to noisy data. We improve the noisy-image classification (NIC) results by significantly large margins, especially at high noise levels, and come close to the fully retrained approaches. Furthermore, as proof of concept, we show that when using our oracle fidelity map we even outperform the fully retrained methods, whether trained on noisy or restored images.

Introduction

Deep learning has revolutionized image classification, primarily through convolutional neural networks (CNNs) that learn rich feature extractors from large datasets. However, these models are often "fair-weather friends," performing brilliantly on clean datasets like ImageNet but failing significantly when faced with real-world noise, such as low-light grain, sensor artifacts, or compression.

Standard solutions to this problem are often inadequate. Preprocessing with a denoiser before classification can fail because restoration processes often introduce artifacts or "smudge" away critical features needed for classification. Conversely, retraining models from scratch on noisy data is computationally expensive and lacks modularity; a new noise type or a better classifier would necessitate a complete restart. This paper proposes the Fidelity-Guided Noisy-Image Classification (FG-NIC) framework, a "plug-and-play" method to make existing, frozen classifiers robust to noise by injecting "fidelity awareness" into their internal layers.

Methodology: The FG-NIC Framework

The core innovation of FG-NIC is its ability to guide a network's attention toward reliable image regions using a "fidelity map." The framework consists of three main components:

  1. Fidelity Map Estimation: The system takes a noisy image, restores it using a network like DnCNN, and then estimates a pixel-wise fidelity map. This map serves as a confidence score (0 to 1) indicating how closely the restored image resembles the original clean version.
  2. Internal Feature Fusion: Instead of simply feeding the restored image to the classifier, FG-NIC fuses the fidelity map into the classifier's internal layers. This is achieved through spatial linear fusion (pixel-by-pixel adjustment), channel fusion (guiding attention across feature channels), and channel concatenation.
  3. Trainable "Sidecar" Modules: While the primary classification network (e.g., ResNet-50) remains frozen, small, lightweight trainable blocks are added to learn the optimal application of fidelity information to the frozen features.
The FG-NIC framework fuses a fidelity map into the internal representations of a frozen feature extractor to guide network attention.

This approach is highly efficient. For a ResNet-50 model, FG-NIC adds only approximately 0.08 G MACs (Multiply-Accumulate operations) to a base requirement of 4.11 G MACs, representing a minimal computational overhead.

Experimental Evaluation

The authors evaluated FG-NIC primarily on the Caltech-256 dataset under various Additive White Gaussian Noise (AWGN) levels ($\sigma$ from 0.1 to 0.5), including spatially varying noise (1D and 2D gradients) to simulate real-world conditions.

Classification accuracy (%) on Caltech-256 across different noise levels and experimental setups.

The results demonstrate that the fidelity map acts as a sophisticated attention mechanism, telling the classifier which parts of the feature space are trustworthy. As shown in the table above, the improvements are most dramatic at high noise levels. At $\sigma = 0.5$, a standard pretrained model's accuracy drops to 1.35%. While standard restoration improves this to 25.53%, the FG-NIC (Pretrained) ensemble reaches 51.56%.

Crucially, the "Oracle" version of FG-NIC, which uses the ground-truth difference between clean and noisy images, achieves 69.51% accuracy at $\sigma = 0.5$. This outperforms even the fully retrained model (66.39%), proving that knowing the spatial distribution of noise is often more valuable than simply training on more noisy examples.

Conclusion

FG-NIC provides a modular and efficient way to enhance the robustness of pretrained classifiers without the need for expensive retraining. By leveraging fidelity estimation to guide internal feature representations, the method significantly closes the performance gap between pretrained and retrained models, especially in high-noise scenarios. While the current performance is tied to the quality of the initial restoration and fidelity estimation, the success of the "Oracle" model suggests that as estimation algorithms improve, this modular framework could become a gold standard for robust noisy-image classification.

Summary updated

Abstract

Image classification has significantly improved using deep learning. This is mainly due to convolutional neural networks (CNNs) that are capable of learning rich feature extractors from large datasets. However, most deep learning classification methods are trained on clean images and are not robust when handling noisy ones, even if a restoration preprocessing step is applied. While novel methods address this problem, they rely on modified feature extractors and thus necessitate retraining. We instead propose a method that can be applied on a $pretrained$ classifier. Our method exploits a fidelity map estimate that is fused into the internal representations of the feature extractor, thereby guiding the attention of the network and making it more robust to noisy data. We improve the noisy-image classification (NIC) results by significantly large margins, especially at high noise levels, and come close to the fully retrained approaches. Furthermore, as proof of concept, we show that when using our oracle fidelity map we even outperform the fully retrained methods, whether trained on noisy or restored images.
BibTeX
@article{lin2021fidelity,
  author = {Xiaoyu Lin and Deblina Bhattacharjee and Majed El Helou and Sabine Süsstrunk},
  title = {{Fidelity Estimation Improves Noisy-Image Classification With Pretrained Networks}},
  journal = {IEEE Signal Processing Letters},
  year = {2021},
  url = {https://openreview.net/forum?id=1vKAvLUxb1-r},
}
Figure