Removing multi-frame Gaussian noise by combining patch-based filters with optical flow

Kireeti Bodduna, Joachim WeickertJournal of Electronic Imaging 2021 · DOI · arXiv

Abstract

Patch-based approaches such as 3D block matching and non-local Bayes are widely accepted filters for removing Gaussian noise from single-frame images. We propose three extensions for these filters when there exist multiple frames of the same scene. The first of them employs reference patches on every frame instead of a commonly used single-reference frame method, thus utilizing the complete available information. The remaining two techniques use a separable spatiotemporal filter to reduce interactions between dissimilar regions, hence mitigating artifacts. In order to deal with non-registered datasets, we combine all our extensions with robust optical flow computation. Two of our proposed multi-frame filters outperform existing extensions on most occasions by a significant margin while also being competitive with a state-of-the-art neural network-based technique. Moreover, one of these two strategies is the fastest among all due to its separable design.

Introduction

Denoising a single image is a well-studied problem, with patch-based algorithms like BM3D (Block-Matching and 3D filtering) and NLB (Non-Local Bayes) serving as gold standards. However, when multiple frames of the same scene are available—such as in video or photo bursts—there is significantly more information to leverage. Existing multi-frame extensions often suffer from under-utilization of data or artifacts like "ghosting" caused by imperfect motion estimation. This paper presents a framework to extend classic single-frame filters into the multi-frame domain efficiently, even in cases of extreme noise and non-registered data.

Proposed Multi-Frame Filtering Strategies

The authors propose and evaluate five different multi-frame extensions. The most significant are the "separable" spatio-temporal filters, which decouple the temporal averaging step from the spatial denoising step.

Characteristics of the proposed and compared multi-frame denoising methods.

Key strategies include:

  • Average then Filter (AF): Frames are registered using optical flow and averaged first. This significantly reduces the noise level (by a factor of $\sqrt{L}$) before a single-frame filter is applied. Adjusting the filter parameters to this new noise level is crucial for performance.
  • Filter then Average (FA): Each registered frame is denoised individually before the results are averaged together.
  • Multiple Reference Frame Filtering (MF): Unlike standard methods that use a single reference frame, this approach treats every frame as a reference during patch-matching.

To handle motion between frames, the authors integrate robust optical flow (variants SOF-1, -2, and -3) designed to preserve motion discontinuities and prevent edge blurring.

Experimental Results

The study utilized both perfectly registered data (e.g., House, Peppers, Bridge) and non-registered sequences (Grove2, Shoe, Bird House) with noise levels ranging from $\sigma = 10$ to $120$.

Visual comparison of denoising results on the Bridge, Peppers, and House datasets.

Key Findings:

  • Superior Accuracy: The BM3D-AF and NLB-AF variants outperformed existing multi-frame extensions (SF, CF) by 1–3 dB PSNR across most noise levels. By separating temporal and spatial steps, these methods are less prone to artifacts from mismatched regions.
  • Competitive with Deep Learning: The proposed methods were competitive with, and sometimes outperformed, the state-of-the-art neural network-based denoiser VNLNET, particularly in registered scenarios.
  • Computational Efficiency: BM3D-AF is approximately 7.25x faster than the more complex BM3D-MF and over 50x faster than NLB-CF on a CPU, making it highly practical for real-time applications.

Conclusion

The separable spatio-temporal filtering approach, particularly the AF strategy, provides a highly effective and efficient framework for multi-frame denoising. While performance relies on robust optical flow for non-registered data, the method's speed and accuracy make it a compelling alternative to complex 3D patch-matching or deep learning approaches. Future work will explore explicit occlusion handling and advanced regularizers to further enhance reconstruction quality.

Summary updated

Abstract

Patch-based approaches such as 3D block matching and non-local Bayes are widely accepted filters for removing Gaussian noise from single-frame images. We propose three extensions for these filters when there exist multiple frames of the same scene. The first of them employs reference patches on every frame instead of a commonly used single-reference frame method, thus utilizing the complete available information. The remaining two techniques use a separable spatiotemporal filter to reduce interactions between dissimilar regions, hence mitigating artifacts. In order to deal with non-registered datasets, we combine all our extensions with robust optical flow computation. Two of our proposed multi-frame filters outperform existing extensions on most occasions by a significant margin while also being competitive with a state-of-the-art neural network-based technique. Moreover, one of these two strategies is the fastest among all due to its separable design.
BibTeX
@article{bodduna2021removing,
  author = {Kireeti Bodduna and Joachim Weickert},
  title = {{Removing multi-frame Gaussian noise by combining patch-based filters with optical flow}},
  journal = {Journal of Electronic Imaging},
  year = {2021},
  doi = {10.1117/1.jei.30.3.033031},
}
Figure