TL;DR
This paper introduces a multi-scale video denoising framework that explicitly refines optical flow estimation under heavy noise conditions. By co-optimizing a task-oriented flow network alongside a mutual feature propagation module, the proposed method achieves state-of-the-art denoising performance on both synthetic and real-world videos while maintaining high computational efficiency.
What Problem Does This Paper Address?
Video denoising relies heavily on temporal information across neighboring frames to distinguish true underlying content from random noise. To align features across frames, state-of-the-art video restoration methods typically utilize optical flow.
However, standard optical flow algorithms break down under high noise levels, leading to severe alignment errors that propagate through the network. This creates a chicken-and-egg problem: you need clean video for accurate flow estimation, but you need accurate flow estimation to produce clean video. Most existing methods ignore this issue and blindly use off-the-shelf flow estimators, leading to sub-optimal denoising and error accumulation.
Methods
The authors propose a multi-scale recurrent network architecture that processes video frames through a down-scaling and up-scaling pipeline. The core innovations lie in two modules:
- Denoising-Oriented Flow Refinement (DFR): Instead of keeping the optical flow network fixed, DFR optimizes a learnable flow estimator (initialized with SpyNet) end-to-end. It minimizes an $\ell_1$ loss between the flows estimated from downscaled noisy frames and pseudo-ground-truth flows derived from noise-free clean frames. This directly trains the flow network to be robust to heavy noise.
- Flow-Guided Mutual Denoising Propagation (FMDP): Using the refined flows, this module simultaneously guides forward and backward feature propagation across multiple scales. The refined flows initialize sampling locations for Deformable Convolutional Networks (DCN), allowing the network to discover meaningful textures and handle noise occlusion dynamically.
Additionally, to bridge the gap between synthetic training and real-world deployment, the authors design a Real Noise Degradation Model. This pipeline randomly shuffles various degradations—including Gaussian noise, Poisson noise, Speckle noise, camera sensor noise, JPEG compression, and video compression artifacts—alongside blur operations.
Datasets
The model was trained and evaluated on several key datasets:
- Synthetic Denoising: Trained on the DAVIS training set (augmented with Additive White Gaussian Noise) and evaluated on both the DAVIS testing set and Set8.
- Real-World Denoising: Evaluated on the CRVD (Camera Raw Video Denoising) dataset, converted to the sRGB domain.
- RealNoise Dataset (New Benchmark): The authors introduced a new dataset collected from Pexels containing 10 diverse real-world scene types (e.g., low-light, underwater, aerial, old videos) exhibiting unknown real-world noise. Because ground-truth clean videos are unavailable for this dataset, performance is measured using non-reference image quality metrics (NIQE, BRISQUE, PIQE).
Key Insights and Ideas
- Flow as a Task-Oriented Component: Treating optical flow estimation as a fixed preprocessing step is a major bottleneck for restoration tasks. Fine-tuning the flow network with a dedicated loss under noisy conditions drastically improves downstream feature alignment.
- Mutual vs. Sequential Propagation: Unlike architectures like BasicVSR++ that propagate forward and backward sequentially (which can accumulate errors over multiple steps), processing mutual propagation across multiple scales aggregates cleaner, more robust temporal context.
- The Importance of Video Compression in Training: Real-world video denoising models often fail because they ignore video compression artifacts during training. Including randomized video compression in the degradation pipeline significantly boosts real-world generalization.
Key Results
- Synthetic Denoising Performance: On the DAVIS dataset under severe noise ($\sigma = 50$), the proposed method outperforms BasicVSR++ by an average of 1.34 dB and the previous state-of-the-art Transformer-based method (VRT) by 0.55 dB PSNR.
- Real Video Denoising: The method achieved the highest PSNR and SSIM across all ISO settings on the CRVD dataset (e.g., 43.38 dB at ISO3200).
- Efficiency Gains: Despite its multi-scale nature, the model is highly efficient. It requires significantly fewer computational resources (182.8G FLOPs) compared to VRT (721.9G FLOPs) while delivering faster inference times and superior denoising quality.
Limitations and Future Work
The authors note a few limitations in their current approach:
- Blur Artifacts: Because the degradation pipeline focuses primarily on noise modeling, the network struggles to remove severe motion or exposure blur artifacts common in real-world video.
- Localized Spot Noise: Large, localized spot noise or severe sensor defects remain challenging to clean completely.
- Artifacts from GAN Loss: When fine-tuned with adversarial loss for real video denoising, the model can occasionally alter fine structural details (such as human faces) under extreme degradation conditions.
Summary updated