DVDNET: A Fast Network for Deep Video Denoising

Matias Tassano, Julie Delon, Thomas Veit

TL;DR

DVDnet is a high-speed, convolutional neural network (CNN) architecture that achieves state-of-the-art video denoising performance. By combining a two-stage spatial-temporal approach with optical flow alignment, it outperforms traditional patch-based methods in both speed (up to 50x faster) and visual quality, particularly at high noise levels.

What Problem Does This Paper Address?

Video denoising is significantly more complex than image denoising because it requires temporal coherence. If you denoise a video frame-by-frame, the slight variations in how noise is removed in each frame lead to "flickering"—a distracting artifact that ruins the viewing experience.

Historically, the "gold standard" for video denoising has been patch-based methods like V-BM4D or VNLB (Video Non-Local Bayes). While effective, these algorithms are computationally expensive, often taking minutes to process a single frame. Previous deep learning attempts struggled to match the quality of these patch-based methods while maintaining reasonable runtimes. DVDnet aims to bridge this gap, providing a fast, memory-efficient CNN that handles a wide range of noise levels with a single model.

Methods

The authors propose a two-stage architecture that separates spatial and temporal processing:

  1. Spatial Denoising Block: Each frame in a sequence (the central frame plus $2T$ neighbors) is first processed individually. This block uses 12 convolutional layers to perform initial denoising.
  2. Alignment (Warping): To handle motion between frames, the pre-denoised neighboring frames are registered to the central frame using DeepFlow, a competitive optical flow algorithm.
  3. Temporal Denoising Block: The aligned frames are concatenated and fed into a 6-layer temporal denoiser. This block "fuses" the information across the frames to ensure temporal consistency and further reduce noise.

Key Technical Features: * Residual Learning: Instead of predicting the clean image, the network predicts the noise (the residual), which is then subtracted from the input. * Noise Map Input: The network takes a "noise map" (an estimate of the noise level $\sigma$) as an additional input. This allows a single trained model to handle varying noise intensities rather than needing a separate model for every possible $\sigma$. * Efficiency: To save on memory and compute, the blocks process data at a quarter resolution and then upscale back to full resolution.

Datasets

  • Spatial Training: The Waterloo Exploration Database was used to extract over 1 million patches for training the spatial denoiser.
  • Temporal Training: The DAVIS database (Densely Annotated VIdeo Segmentation) provided the sequences for training the temporal fusion block.
  • Evaluation: Benchmarking was performed on the DAVIS-test set and Set8 (a collection of sequences from GoPro and the Derfs Test Media).

Key Insights and Ideas

  • Denoise Before Aligning: One of the cleverest moves in DVDnet is performing spatial denoising before optical flow estimation. Optical flow is notoriously sensitive to noise; by cleaning the frames first, the alignment becomes much more accurate.
  • The "Over-Denoising" Trade-off: The authors observed that while their model excels at high noise levels, traditional methods like VNLB still hold an edge at very low noise levels ($\sigma=10$). DVDnet tends to be slightly too aggressive in these cases, smoothing out fine textures.
  • Bottleneck Identification: Interestingly, the CNN itself is extremely fast. Out of the 8 seconds it takes to process a frame, 6 seconds are spent on the DeepFlow algorithm. This highlights that the future of fast video denoising lies as much in faster motion estimation as it does in better neural architectures.

Key Results

  • Speed: DVDnet takes less than 8 seconds to denoise a 960x540 color frame on a GPU. This is roughly 20x faster than V-BM4D and 50x faster than VNLB.
  • Quality: On the Set8 and DAVIS datasets, DVDnet surpasses VNLB in PSNR (Peak Signal-to-Noise Ratio) once noise levels exceed $\sigma=30$.
  • Visuals: Qualitatively, DVDnet produces much smoother results in flat areas (like skies or walls) where patch-based methods often leave behind "low-frequency" blotchy noise. It also shows remarkable temporal stability with almost no visible flickering.

Limitations and Future Work

  • Gaussian Assumption: The current model is trained and tested on Additive White Gaussian Noise (AWGN). While the authors suggest it can be extended to other noise types (like Poisson noise), this wasn't the primary focus of the paper.
  • Low-Noise Performance: As noted, the model can "over-denoise" at low noise levels, losing some micro-textures that VNLB manages to preserve.
  • Flow Dependency: The system's speed and accuracy are heavily dependent on the external optical flow algorithm (DeepFlow). Integrating a faster, end-to-end trainable flow estimation could be a logical next step.

Summary updated

Abstract

In this paper, we propose a state-of-the-art video denoising algorithm based on a convolutional neural network architecture. Previous neural network based approaches to video denoising have been unsuccessful as their performance cannot compete with the performance of patch-based methods. However, our approach outperforms other patch-based competitors with significantly lower computing times. In contrast to other existing neural network denoisers, our algorithm exhibits several desirable properties such as a small memory footprint, and the ability to handle a wide range of noise levels with a single network model. The combination between its denoising performance and lower computational load makes this algorithm attractive for practical denoising applications. We compare our method with different state-of-art algorithms, both visually and with respect to objective quality metrics. The experiments show that our algorithm compares favorably to other state-of-art methods. Video examples, code and models are publicly available at https://github.com/m-tassano/dvdnet.
BibTeX
@inproceedings{tassano2019dvdnet,
  author = {Matias Tassano and Julie Delon and Thomas Veit},
  title = {{DVDNET: A Fast Network for Deep Video Denoising}},
  booktitle = {ICIP},
  year = {2019},
  url = {https://openreview.net/forum?id=GBZvIUimQz},
}

Loading related content...

Figure