Fast, Trainable, Multiscale Denoising

Sungjoon Choi, John Isidoro, Pascal Getreuer, Peyman Milanfar

Abstract

Denoising is a fundamental imaging problem. Versatile but fast filtering has been demanded for mobile camera systems. We present an approach to multiscale filtering which allows real-time applications on low-powered devices. The key idea is to learn a set of kernels that upscales, filters, and blends patches of different scales guided by local structure analysis. This approach is trainable so that learned filters are capable of treating diverse noise patterns and artifacts. Experimental results show that the presented approach produces comparable results to state-of-the-art algorithms while processing time is orders of magnitude faster.

Introduction

Image denoising is a fundamental challenge in computer vision. While modern methods like BM3D or deep Convolutional Neural Networks (CNNs) produce high-quality results, their computational weight often makes them unsuitable for real-time applications on mobile cameras and low-power devices. Removing heavy noise typically requires large receptive fields, which traditionally leads to a quadratic increase in processing time. This paper addresses this gap by introducing a multiscale denoising algorithm that achieves a broad receptive field using small, efficient filters, enabling high-performance denoising at speeds orders of magnitude faster than current state-of-the-art techniques.

Methodology

The proposed method extends the "Best Linear Adaptive Enhancement" (BLADE) framework into a multiscale pipeline. The architecture relies on spatially-adaptive filtering, which is highly parallelizable and hardware-friendly.

Spatially-Adaptive Filtering

The algorithm analyzes the local neighborhood of every pixel to compute a structure tensor, determining local gradient orientation, strength, and coherence. Notably, this analysis is performed jointly across all RGB channels to preserve color-specific edges that might be missed in luminance-only processing. Based on these structural features, a small linear filter (e.g., 5x5) is selected from a pre-trained filterbank and applied to the patch.

The BLADE framework: An input patch is analyzed for filter selection from a pre-trained bank to produce an output pixel.

Multiscale Cascade

To handle large-scale noise without the overhead of large filters, the algorithm creates an image pyramid. Starting from the coarsest resolution, learned filters are applied that simultaneously upscale the coarser image, denoise the current level, and blend them together. This cascading approach allows a small 5x5 filter to effectively cover a much larger area of the original image (e.g., 20x20 pixels after two downsampling steps), providing the necessary context for heavy noise removal at a fraction of the cost.

The multiscale cascade: Learned kernels jointly upscale, filter, and blend image patches across resolution levels.

Experimental Results

The filterbanks were trained on a dataset of 2.23 billion pixels from high-quality internet images, augmented with synthetic noise and geometric transformations. Evaluation was performed using the Berkeley dataset.

Speed and Efficiency

The standout feature of this approach is its processing speed. On a 1-megapixel image, the method takes just 0.038 seconds, representing a 400x to 500x speedup over methods like C-BM3D (21.9s) and TNRD (16.5s). On a 2017 Google Pixel phone, the GPU implementation achieves a throughput of 188 Megapixels per second (MP/s), making it highly viable for real-time mobile photography.

Quality Trade-offs

The algorithm achieves competitive image quality. At a noise standard deviation of $\sigma=25$, the method reaches a PSNR of 29.58 dB, which is close to the 30.18 dB achieved by the much slower C-BM3D. This small quality difference is a favorable trade-off for the massive gains in efficiency.

Key Insights and Conclusion

The efficiency of this method stems from its "shallow" architecture, which utilizes simple additions and multiplications with few sequential dependencies. This makes it ideal for hardware vectorization and parallelization compared to deep CNNs. Furthermore, joint RGB analysis ensures that structural details are preserved across all color channels.

Future work will focus on integrating automatic noise estimation to handle non-uniform noise levels and developing sensor-specific models to further optimize performance for individual camera hardwares.

Summary updated

Abstract

Denoising is a fundamental imaging problem. Versatile but fast filtering has been demanded for mobile camera systems. We present an approach to multiscale filtering which allows real-time applications on low-powered devices. The key idea is to learn a set of kernels that upscales, filters, and blends patches of different scales guided by local structure analysis. This approach is trainable so that learned filters are capable of treating diverse noise patterns and artifacts. Experimental results show that the presented approach produces comparable results to state-of-the-art algorithms while processing time is orders of magnitude faster.
BibTeX
@inproceedings{choi2018fast,
  author = {Sungjoon Choi and John Isidoro and Pascal Getreuer and Peyman Milanfar},
  title = {{Fast, Trainable, Multiscale Denoising}},
  booktitle = {ICIP},
  year = {2018},
  url = {https://openreview.net/forum?id=gypl48WFF3F},
}
Figure