Abstract
Patch-based approaches such as 3D block matching and non-local Bayes are widely accepted filters for removing Gaussian noise from single-frame images. We propose three extensions for these filters when there exist multiple frames of the same scene. The first of them employs reference patches on every frame instead of a commonly used single-reference frame method, thus utilizing the complete available information. The remaining two techniques use a separable spatiotemporal filter to reduce interactions between dissimilar regions, hence mitigating artifacts. In order to deal with non-registered datasets, we combine all our extensions with robust optical flow computation. Two of our proposed multi-frame filters outperform existing extensions on most occasions by a significant margin while also being competitive with a state-of-the-art neural network-based technique. Moreover, one of these two strategies is the fastest among all due to its separable design.
Introduction
Denoising a single image is a well-studied problem, with patch-based algorithms like BM3D (Block-Matching and 3D filtering) and NLB (Non-Local Bayes) serving as gold standards. However, when multiple frames of the same scene are available—such as in video or photo bursts—there is significantly more information to leverage. Existing multi-frame extensions often suffer from under-utilization of data or artifacts like "ghosting" caused by imperfect motion estimation. This paper presents a framework to extend classic single-frame filters into the multi-frame domain efficiently, even in cases of extreme noise and non-registered data.
Proposed Multi-Frame Filtering Strategies
The authors propose and evaluate five different multi-frame extensions. The most significant are the "separable" spatio-temporal filters, which decouple the temporal averaging step from the spatial denoising step.
Key strategies include:
- Average then Filter (AF): Frames are registered using optical flow and averaged first. This significantly reduces the noise level (by a factor of $\sqrt{L}$) before a single-frame filter is applied. Adjusting the filter parameters to this new noise level is crucial for performance.
- Filter then Average (FA): Each registered frame is denoised individually before the results are averaged together.
- Multiple Reference Frame Filtering (MF): Unlike standard methods that use a single reference frame, this approach treats every frame as a reference during patch-matching.
To handle motion between frames, the authors integrate robust optical flow (variants SOF-1, -2, and -3) designed to preserve motion discontinuities and prevent edge blurring.
Experimental Results
The study utilized both perfectly registered data (e.g., House, Peppers, Bridge) and non-registered sequences (Grove2, Shoe, Bird House) with noise levels ranging from $\sigma = 10$ to $120$.
Key Findings:
- Superior Accuracy: The BM3D-AF and NLB-AF variants outperformed existing multi-frame extensions (SF, CF) by 1–3 dB PSNR across most noise levels. By separating temporal and spatial steps, these methods are less prone to artifacts from mismatched regions.
- Competitive with Deep Learning: The proposed methods were competitive with, and sometimes outperformed, the state-of-the-art neural network-based denoiser VNLNET, particularly in registered scenarios.
- Computational Efficiency: BM3D-AF is approximately 7.25x faster than the more complex BM3D-MF and over 50x faster than NLB-CF on a CPU, making it highly practical for real-time applications.
Conclusion
The separable spatio-temporal filtering approach, particularly the AF strategy, provides a highly effective and efficient framework for multi-frame denoising. While performance relies on robust optical flow for non-registered data, the method's speed and accuracy make it a compelling alternative to complex 3D patch-matching or deep learning approaches. Future work will explore explicit occlusion handling and advanced regularizers to further enhance reconstruction quality.
Summary updated