Fidelity Estimation Improves Noisy-Image Classification With Pretrained Networks

Xiaoyu Lin, Deblina Bhattacharjee, Majed El Helou, Sabine Süsstrunk

Abstract

Image classification has significantly improved using deep learning, primarily due to convolutional neural networks (CNNs) capable of learning rich feature extractors from large datasets. However, most deep learning classification methods are trained on clean images and are not robust when handling noisy ones, even if a restoration preprocessing step is applied. While novel methods address this problem, they rely on modified feature extractors and thus necessitate retraining. We instead propose a method that can be applied on a pretrained classifier. Our method exploits a fidelity map estimate that is fused into the internal representations of the feature extractor, thereby guiding the attention of the network and making it more robust to noisy data. We improve the noisy-image classification (NIC) results by significantly large margins, especially at high noise levels, and come close to the fully retrained approaches. Furthermore, as proof of concept, we show that when using our oracle fidelity map we even outperform the fully retrained methods, whether trained on noisy or restored images.

Introduction: The Robustness Gap

Deep learning has revolutionized image classification, yet state-of-the-art Convolutional Neural Networks (CNNs) often exhibit a "glass jaw" when faced with noise. Most models are trained on pristine datasets like ImageNet, and their accuracy drops precipitously when deployed in real-world environments characterized by low-light conditions, medical imaging artifacts, or sensor noise.

Standard solutions typically fall into two categories: restoration preprocessing and full retraining. Restoration often fails because denoisers can "smear" critical textures required for classification. Retraining, while effective, is computationally expensive and discards the benefits of high-quality, "off-the-shelf" pretrained models. The proposed FG-NIC (Fidelity-Guided Noisy-Image Classification) framework addresses this by "rescuing" fixed, pretrained classifiers without requiring a full architectural overhaul.

Methodology: The FG-NIC Framework

The core of the FG-NIC approach is the integration of a Fidelity Map into the feature extraction process. The pipeline consists of three main stages:

  1. Restoration: A noisy input image $\mathbf{X}$ is first processed by a standard denoiser (e.g., DnCNN) to produce a restored image $\tilde{\mathbf{X}}$.
  2. Fidelity Estimation: A small, dedicated network estimates a fidelity map $\mathbf{F}$, which serves as a pixel-wise confidence score indicating which parts of the restored image are most reliable.
  3. Fidelity-Guided Fusion: This map is used to modulate the internal features of a frozen backbone classifier (e.g., ResNet-50).
Overview of the FG-NIC pipeline showing how the fidelity map guides the feature extractor through spatial and channel fusion.

Crucially, the main backbone remains frozen. Only small fusion modules are trained to perform Spatial Fusion (directing the network where to look) and Channel Fusion (determining which feature channels to trust). This modularity allows the method to be plugged into existing architectures with minimal overhead.

Core Innovation: The Fidelity Map

In an ideal scenario, the fidelity map represents the difference between a noisy image and its clean counterpart. In practice, FG-NIC trains a small network to predict this map from the noisy input. By treating the error map of a denoiser as an attention mechanism, the system can "ignore" artifacts where the denoiser likely failed.

Visual comparison of clean and degraded images alongside their estimated fidelity maps across various noise levels (σ) and patterns (1D/2D varying).

As shown in the visual examples, the fidelity map captures the structural integrity of the image. The authors also tested an "Oracle" version using ground-truth fidelity maps, which surprisingly outperformed models fully retrained on noisy data. This suggests that knowing the location and intensity of noise is more powerful for a classifier than simply being exposed to more noisy training examples.

Experimental Results

The method was evaluated on Caltech-256 and Caltech-101 using Additive White Gaussian Noise (AWGN) and spatially varying noise. The improvements are most dramatic at high noise levels ($\sigma = 0.5$):

  • Baseline Failure: A standard ResNet-50 trained on clean data drops to 1.35% accuracy on noisy images.
  • FG-NIC Performance: The FG-NIC (pretrained) approach restores this accuracy to 56.44%.
  • Efficiency: The fusion modules add only 0.08 Billion MACs to a ResNet-50, representing negligible computational cost compared to the backbone itself.

Discussion and Future Work

FG-NIC demonstrates that modular robustness is achievable without the high cost of retraining. While the current study focuses on Gaussian noise, the framework's success with spatially varying noise suggests potential for handling other artifacts like motion blur or compression. Future work may explore "black-box" versions of this method that do not require access to the internal layers of the CNN backbone, further increasing its applicability to proprietary models.

Summary updated

Abstract

Image classification has significantly improved using deep learning. This is mainly due to convolutional neural networks (CNNs) that are capable of learning rich feature extractors from large datasets. However, most deep learning classification methods are trained on clean images and are not robust when handling noisy ones, even if a restoration preprocessing step is applied. While novel methods address this problem, they rely on modified feature extractors and thus necessitate retraining. We instead propose a method that can be applied on a pretrained classifier. Our method exploits a fidelity map estimate that is fused into the internal representations of the feature extractor, thereby guiding the attention of the network and making it more robust to noisy data. We improve the noisy-image classification (NIC) results by significantly large margins, especially at high noise levels, and come close to the fully retrained approaches. Furthermore, as proof of concept, we show that when using our oracle fidelity map we even outperform the fully retrained methods, whether trained on noisy or restored images.
BibTeX
@article{lin2021fidelity,
  author = {Xiaoyu Lin and Deblina Bhattacharjee and Majed El Helou and Sabine Süsstrunk},
  title = {{Fidelity Estimation Improves Noisy-Image Classification With Pretrained Networks}},
  journal = {IEEE Signal Processing Letters},
  year = {2021},
  url = {https://openreview.net/forum?id=1vKAvLUxb1-r},
}
Figure