On the Importance of Denoising When Learning To Compress Images

Benoit Brummer, Christophe De VleeschouwerWACV 2023

TL;DR

This paper introduces a novel approach to image compression by jointly training a single neural network to both denoise and compress images. This integrated method significantly improves rate-distortion performance on noisy images compared to traditional methods or separate denoising and compression steps, while also being computationally more efficient.

What Problem Does This Paper Address?

Image sensors inherently capture noise alongside the actual image content. This noise is undesirable and, crucially, incompressible. Standard lossy image compressors, even learned ones, often struggle with noisy inputs, leading to inflated file sizes and reduced visual quality. While denoising an image before compression can improve results, it adds complexity and computational overhead. This paper addresses the challenge of achieving high-quality, efficient image compression on noisy images without resorting to a multi-stage pipeline.

Methods

The core contribution is a new supervision strategy for training a joint denoising and compression autoencoder. Instead of training a separate denoiser and then a compressor, or relying on implicit denoising from perception-based loss functions, this work trains a single network using paired noisy and clean images. The network is tasked with reconstructing the clean image from a potentially noisy input, effectively learning to denoise and compress simultaneously.

The authors propose several supervision strategies:

  • JDC-N: Training with only noisy-clean pairs.
  • JDC-CN: Incorporating clean-clean pairs to maintain performance on clean images.
  • JDC-Cn: Filtering training data to focus on specific noise levels (e.g., MS-SSIM ≥ 0.8), finding that extreme denoising can be detrimental.
  • JDC-UD: Using knowledge distillation from a pre-trained universal denoiser.

The proposed "Natural Image Noise Removal" (NINR) supervision, particularly the JDC-Cn approach with a quality threshold, proved most effective.

Datasets

  • Natural Image Noise Dataset (NIND) [8]: This dataset, containing various scenes with different ISO settings and corresponding clean ground-truth images, is central to the training and evaluation.
  • Wikimedia Commons Featured Pictures [1]: Used to obtain high-quality clean images for training and evaluation, ensuring the model doesn't degrade on noise-free content.
  • CLIC professional test set [2]: Used for evaluating models on clean images.

Key Insights and Ideas

  • Joint Denoising and Compression is Key: Explicitly training a single network for both tasks yields superior rate-distortion performance on noisy images compared to separate denoising and compression.
  • The Peril of Extreme Denoising: Training a model to perform overly aggressive denoising can hurt overall rate-distortion, suggesting a balance is needed. Focusing on realistic noise levels (e.g., MS-SSIM ≥ 0.8) is more beneficial.
  • Clean Images Matter: Including clean-clean image pairs in the training set is crucial for maintaining competitive performance on noise-free images.
  • Computational Efficiency: A single joint model is significantly more computationally efficient (e.g., 10.3% of the complexity of a separate denoiser and compressor) while achieving better results.

Key Results

  • The proposed joint denoising and compression (JDC) models, particularly JDC-Cn.8, achieve significantly better rate-distortion performance on noisy images compared to standard compression methods (JPEG, JPEG XL, BPG) and even a standard learned autoencoder.
  • The JDC models outperform a two-step approach (denoising then compression) by a large margin in terms of computational complexity (e.g., 8.9 times fewer GMac operations than a U-Net denoiser followed by a compression encoder).
  • The JDC-Cn.8 model, trained with a quality threshold of MS-SSIM ≥ 0.8, shows results comparable to or better than standard models on clean images and substantially better on noisy images.
  • Visualizations (Figure 1, 6, 7) clearly demonstrate the superior ability of the JDC models to reconstruct clean details from noisy inputs, avoiding artifacts seen in other methods.

Limitations and Future Work

  • Noise Threshold Sensitivity: The paper notes that the optimal noise threshold for training (e.g., MS-SSIM ≥ 0.8) might depend on the specific dataset and application. Training with too high a threshold (e.g., ≥ 0.9) can lead to worse performance on noisier test images.
  • Knowledge Distillation Trade-offs: While JDC-UD (knowledge distillation) offers a virtually limitless training dataset, it results in slightly worse rate-distortion compared to training with curated noisy-clean pairs.
  • Future Work: The authors suggest that their NINR supervision strategy is fundamental and generic, applicable to any compression architecture. Future work could explore its integration with more advanced compression models and investigate optimal noise threshold selection strategies.

Summary updated

BibTeX
@inproceedings{brummer2023importance,
  author = {Benoit Brummer and Christophe De Vleeschouwer},
  title = {{On the Importance of Denoising When Learning To Compress Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure