DVDnet: A Fast Network for Deep Video Denoising

Matias Tassano, Julie Delon, Thomas Veit

Abstract

In this paper, we propose a state-of-the-art video denoising algorithm based on a convolutional neural network architecture. Previous neural network based approaches to video denoising have been unsuccessful as their performance cannot compete with the performance of patch-based methods. However, our approach outperforms other patch-based competitors with significantly lower computing times. In contrast to other existing neural network denoisers, our algorithm exhibits several desirable properties such as a small memory footprint, and the ability to handle a wide range of noise levels with a single network model. The combination between its denoising performance and lower computational load makes this algorithm attractive for practical denoising applications. We compare our method with different state-of-art algorithms, both visually and with respect to objective quality metrics. The experiments show that our algorithm compares favorably to other state-of-art methods. Video examples, code and models are publicly available at https://github.com/m-tassano/dvdnet.

Introduction

Video denoising is a critical task in computer vision, complicated by the need for temporal coherence. Unlike image denoising, processing video frame-by-frame often results in "flickering" artifacts due to inconsistent noise removal across the temporal dimension. Historically, patch-based methods like V-BM4D and VNLB (Video Non-Local Bayes) have defined the state-of-the-art. However, these methods are computationally intensive, often requiring minutes to process a single frame.

DVDnet addresses these challenges by providing a fast, memory-efficient convolutional neural network (CNN) that achieves superior performance while maintaining significantly lower runtimes. It is designed to handle a broad spectrum of noise levels using a single model, making it highly practical for real-world applications.

Proposed Architecture

DVDnet utilizes a two-stage architecture that decouples spatial and temporal processing to ensure both high-quality denoising and temporal stability.

The DVDnet architecture: a two-stage process involving spatial denoising, warping for alignment, and temporal fusion.

Spatial Denoising and Alignment

The process begins with a Spatial Denoising Block. Each frame in a sequence (the central frame $t$ and its $2T$ neighbors) is processed individually. A key insight of DVDnet is performing denoising before motion compensation. Since optical flow estimation is sensitive to noise, pre-denoising the frames allows for more accurate alignment. Following spatial denoising, neighboring frames are registered to the central frame using the DeepFlow algorithm.

Temporal Fusion

The aligned frames are then passed to the Temporal Denoising Block. This stage fuses information across the temporal window to reduce remaining noise and ensure consistency, effectively eliminating flickering.

Detailed structure of the spatial (top) and temporal (bottom) denoising blocks, featuring residual learning and noise map inputs.

Technical Implementation

  • Residual Learning: The network is trained to predict the noise component (the residual) rather than the clean image. This residual is then subtracted from the input to produce the denoised output.
  • Noise Map Input: To allow a single model to generalize across different noise intensities, a "noise map" (an estimate of the noise level $\sigma$) is provided as an additional input to the network.
  • Efficiency: The architecture processes data at a reduced resolution to minimize memory and computational overhead, upscaling back to full resolution for the final output.

Experimental Results

DVDnet was evaluated using the Waterloo Exploration Database for spatial training and the DAVIS database for temporal training. Benchmarking on Set8 and DAVIS-test demonstrated significant improvements over traditional methods.

  • Performance: DVDnet surpasses VNLB in PSNR once noise levels exceed $\sigma=30$. Qualitatively, it produces smoother results in flat regions and maintains excellent temporal stability.
  • Speed: The algorithm processes a 960x540 color frame in less than 8 seconds on a GPU, making it approximately 20x faster than V-BM4D and 50x faster than VNLB. Notably, the majority of this time (6 seconds) is consumed by the DeepFlow alignment, suggesting that the CNN components themselves are extremely efficient.

Conclusion and Future Work

DVDnet represents a significant step forward in deep video denoising, matching or exceeding the quality of patch-based methods at a fraction of the computational cost. While the model excels at high noise levels, it can occasionally "over-denoise" at very low noise levels ($\sigma=10$), leading to a loss of fine textures. Future improvements could involve integrating faster, end-to-end trainable flow estimation to further reduce processing times and extending the model to handle non-Gaussian noise types.

Summary updated

Abstract

In this paper, we propose a state-of-the-art video denoising algorithm based on a convolutional neural network architecture. Previous neural network based approaches to video denoising have been unsuccessful as their performance cannot compete with the performance of patch-based methods. However, our approach outperforms other patch-based competitors with significantly lower computing times. In contrast to other existing neural network denoisers, our algorithm exhibits several desirable properties such as a small memory footprint, and the ability to handle a wide range of noise levels with a single network model. The combination between its denoising performance and lower computational load makes this algorithm attractive for practical denoising applications. We compare our method with different state-of-art algorithms, both visually and with respect to objective quality metrics. The experiments show that our algorithm compares favorably to other state-of-art methods. Video examples, code and models are publicly available at \url{https://github.com/m-tassano/dvdnet}.
BibTeX
@inproceedings{tassano2019dvdnet,
  author = {Matias Tassano and Julie Delon and Thomas Veit},
  title = {{DVDnet: A Fast Network for Deep Video Denoising}},
  booktitle = {ICIP},
  year = {2019},
  url = {https://openreview.net/forum?id=GBZvIUimQz},
}
Figure