Abstract
Video denoising aims at removing noise from videos to recover clean ones. Some existing works show that optical flow can help the denoising by exploiting the additional spatial-temporal clues from nearby frames. However, the flow estimation itself is also sensitive to noise, and can be unusable under large noise levels. To this end, we propose a new multi-scale refined optical flow-guided video denoising method, which is more robust to different noise levels. Our method mainly consists of a denoising-oriented flow refinement (DFR) module and a flow-guided mutual denoising propagation (FMDP) module. Unlike previous works that directly use off-the-shelf flow solutions, DFR first learns robust multi-scale optical flows, and FMDP makes use of the flow guidance by progressively introducing and refining more flow information from low resolution to high resolution. Together with real noise degradation synthesis, the proposed multi-scale flow-guided denoising network achieves state-of-the-art performance on both synthetic Gaussian denoising and real video denoising.
Introduction
Video denoising is significantly more complex than image denoising due to the requirement for temporal alignment. To effectively clean a single frame, models must aggregate information from neighboring frames by identifying matching pixels. This alignment is typically facilitated by optical flow. However, a significant "catch-22" exists: standard optical flow algorithms (e.g., SpyNet) are designed for clean imagery. Under heavy noise degradation, flow estimation often fails, leading to "ghosting" artifacts or inadequate noise removal.
Most existing methods treat flow as a static, "black box" input. This research addresses the robustness gap by proposing a task-oriented approach where flow estimation is refined specifically for the denoising objective, ensuring reliable guidance even in extreme noise conditions.
Methodology
The proposed architecture utilizes a multi-scale recurrent network featuring two primary innovations:
1. Denoising-oriented Flow Refinement (DFR)
Rather than relying on off-the-shelf flow, the DFR module optimizes the flow estimation network end-to-end. It specifically minimizes the discrepancy between flows derived from noisy frames and those from clean ground-truth frames. This "task-oriented" learning enables the model to ignore noise and focus on the underlying motion.
2. Flow-guided Mutual Denoising Propagation (FMDP)
This module employs the refined flows to guide feature alignment via Deformable Convolutional Networks (DCN). The refined flow provides an initial "starting point" for DCN offsets, which are then further tuned to capture relevant textures. Unlike sequential propagation, FMDP allows forward and backward features to mutually guide each other across multiple scales.
Real Noise Degradation Model
To improve real-world performance, the authors developed a sophisticated degradation pipeline. It moves beyond simple Gaussian noise to include sensor noise, Poisson noise, and compression artifacts, shuffled to create realistic training data.
Experimental Results
The model was evaluated on synthetic benchmarks (DAVIS, Set8), real-world raw data (CRVD), and a new "in-the-wild" benchmark called RealNoise, featuring diverse scenes like underwater and aerial footage.
Quantitative and Qualitative Performance
On the DAVIS dataset, the proposed method outperformed the previous state-of-the-art (VRT) by 0.55dB and BasicVSR++ by 1.34dB. Notably, the performance gap widens as noise levels increase, demonstrating the robustness of the flow refinement module. Qualitatively, the model preserves fine details—such as small text or complex textures—that competing methods often blur or fail to recover.
Efficiency
While Transformer-based models like VRT achieve high performance, they are computationally intensive. This recurrent multi-scale approach achieves superior results with significantly fewer FLOPs and faster inference times, as shown in the performance-runtime analysis where the proposed model occupies the top-left quadrant (high PSNR, low runtime).
Conclusion
This work demonstrates that optical flow should be a learnable, task-oriented feature rather than a static input for video denoising. By refining flow to "see through" noise, the model achieves state-of-the-art results with high efficiency. Future work will address limitations such as motion blur artifacts from long exposures and the preservation of identity in extremely degraded facial videos.
Summary updated