Fast, Trainable, Multiscale Denoising

Sungjoon Choi, John Isidoro, Pascal Getreuer, Peyman Milanfar

TL;DR

This paper introduces a lightning-fast, trainable multiscale denoising algorithm designed specifically for low-power devices like mobile phone cameras. By learning a set of small filters that jointly upscale, denoise, and blend image patches across different resolution scales, the method achieves image quality comparable to state-of-the-art techniques but runs orders of magnitude faster.

What Problem Does This Paper Address?

Image denoising is a fundamental and decades-old challenge in computer vision. While modern methods like patch-matching (e.g., BM3D) or deep Convolutional Neural Networks (CNNs) produce incredibly high-quality, noise-free images, they are computationally heavy.

For mobile cameras and low-power devices, these state-of-the-art methods are simply too slow and battery-intensive for real-time applications. If you want to remove heavy noise, you typically need larger filter sizes or deeper networks to understand the broader context of the image, which quadratically increases processing time. The authors tackle this gap by asking: How can we achieve the broad receptive field needed for heavy noise removal without the crippling computational cost of large filters or deep networks?

Methods

The authors build upon a "shallow" learning framework called BLADE (Best Linear Adaptive Enhancement) and extend it into a multiscale pipeline.

Instead of passing an image through dozens of neural network layers, this method uses spatially-adaptive filtering. Here is how it works:

  1. Structure Analysis: For every pixel, the algorithm looks at its local neighborhood and computes a "structure tensor" to determine the local gradient's orientation, strength, and coherence (how edge-like it is). Notably, it computes this jointly across all RGB color channels, rather than just grayscale (luma), to avoid missing color-specific edges.
  2. Filter Selection: Based on these three structural features, the algorithm selects a single, small linear filter (e.g., 5x5) from a pre-trained filterbank.
  3. Multiscale Cascade: To handle large noise patterns without using slow, large filters, the algorithm creates an image pyramid by downsampling the noisy image by factors of two. Starting from the smallest, coarsest image, it applies learned filters that simultaneously upscale the coarser image, denoise the current finer level, and blend them together.

Because the inference step only requires calculating local gradients, looking up a filter, and applying a simple linear dot product, it is highly parallelizable and incredibly fast.

Datasets

  • Training: The filterbanks were trained on high-quality images collected from the internet (consuming about 2.23 billion pixels). These were augmented with synthetic Additive White Gaussian Noise (AWGN), spatial flips, and rotations.
  • Evaluation: The method was tested on 68 standard test images from the well-known Berkeley dataset.

Key Insights and Ideas

  • Color matters for edge detection: Many fast image processing algorithms only analyze the luma (brightness) channel to find edges and save compute. The authors found this causes blurry reconstructions because some structures are only visible in color. Joint RGB structure analysis solves this with minimal overhead.
  • Multiscale pyramids simulate massive filters: A 5x5 filter applied to an image that has been downsampled twice effectively covers a 20x20 pixel area of the original image. By cascading these small filters up a resolution pyramid, the algorithm successfully removes heavy noise that would normally require massive, computationally expensive kernels.
  • Shallow networks win on mobile: Because the architecture relies on simple additions and multiplications on sequential data with few dependencies, it is incredibly friendly to hardware vectorization and parallelization (unlike deep CNNs which require heavy matrix multiplications and memory bandwidth).

Key Results

The standout result of this paper is its sheer speed:

  • Desktop Speedup: On a 1-megapixel image, the proposed method takes just 0.038 seconds. For comparison, state-of-the-art methods like C-BM3D took 21.9 seconds, and TNRD took 16.5 seconds. This represents roughly a 400x to 500x speedup.
  • Mobile Performance: On a 2017 Google Pixel phone, inference runs at 18 Megapixels per second (MP/s) on the CPU and an impressive 188 MP/s on the GPU, making it highly viable for real-time mobile photography.
  • Quality: The algorithm trades a tiny fraction of quality for this massive speed gain. At a noise standard deviation of $\sigma=25$, the method achieves a PSNR of 29.58 dB, which is highly competitive with heavy-duty algorithms like C-BM3D (30.18 dB).

Limitations and Future Work

The authors acknowledge a few limitations in their current pipeline. Primarily, the system assumes that the noise level of the input image is both known and uniform across all pixels, which is rarely true in real-world mobile photography.

For future work, they suggest integrating automatic noise estimation techniques to dynamically guide the filter selection process. Additionally, they propose characterizing the specific noise response models of individual camera sensors to train bespoke filterbanks, which would further improve real-world denoising performance.

Summary updated

Abstract

Denoising is a fundamental imaging problem. Versatile but fast filtering has been demanded for mobile camera systems. We present an approach to multiscale filtering which allows real-time applications on low-powered devices. The key idea is to learn a set of kernels that upscales, filters, and blends patches of different scales guided by local structure analysis. This approach is trainable so that learned filters are capable of treating diverse noise patterns and artifacts. Experimental results show that the presented approach produces comparable results to state-of-the-art algorithms while processing time is orders of magnitude faster.
BibTeX
@inproceedings{choi2018fast,
  author = {Sungjoon Choi and John Isidoro and Pascal Getreuer and Peyman Milanfar},
  title = {{Fast, Trainable, Multiscale Denoising}},
  booktitle = {ICIP},
  year = {2018},
  url = {https://openreview.net/forum?id=gypl48WFF3F},
}

Loading related content...

Figure