Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution

Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, Jiaya Jia

TL;DR

Researchers have developed a novel framework called Deep Parametric 3D Filters (DP3DF) that simultaneously performs video super-resolution, denoising, and illumination enhancement in a single, efficient step. By unifying these three distinct tasks into one mathematical representation, the model produces sharper, brighter, and cleaner high-resolution videos from low-light, noisy inputs faster than traditional multi-stage pipelines.

What Problem Does This Paper Address?

Capturing high-quality video in low-light environments is a "triple threat" for computer vision. The resulting footage is typically low-resolution, buried in sensor noise, and underexposed.

Standard Video Super-Resolution (SR) models are designed for "clean" inputs and often fail miserably when faced with heavy noise or darkness. To solve this, practitioners usually "cascade" models: first run a denoiser, then an illumination enhancer, and finally an SR model. However, this approach has three major flaws:

  1. Inconsistency: Artifacts from the first model (like blurring from denoising) get amplified by the subsequent SR model.
  2. Complexity: Running three separate deep networks is computationally expensive and slow.
  3. Information Loss: Each stage produces a single-frame output, stripping away the rich temporal context (information from adjacent frames) that a subsequent stage might need.

Methods

The authors introduce the Deep Parametric 3D Filter (DP3DF). Instead of predicting pixels directly, the network learns to predict filters that are applied to the input video.

The 3D Filter Magic

Unlike standard 2D filters that only look at a single image, DP3DF looks at a 3D spatiotemporal volume (height, width, and time).

  • Denoising & SR: The filter weights are normalized (using Softmax) to perform local smoothing, which suppresses noise while upsampling the resolution.
  • Illumination: The authors added a clever "+1" dimension to the filter. This extra parameter acts as a learnable gain for luminance adjustment, brightening dark areas based on the local context.

The Architecture

The system uses a single encoder-decoder backbone with two specialized branches:

  1. Filter Learning Branch: Predicts the DP3DF coefficients for every pixel.
  2. Residual Learning Branch: Learns a "residual image" to recover fine, high-frequency textures that might be lost during the filtering process.

By sharing the backbone, the model remains lightweight and ensures that the denoising and brightening efforts are physically "aware" of the super-resolution task.

Datasets

The model was trained and evaluated on two primary real-world datasets:

  • SMID (See-In-The-Dark): Contains static low-light videos paired with long-exposure ground truths.
  • SDSD (Static and Dynamic Scene Dataset): A more challenging dataset containing both indoor and outdoor scenes with moving objects, captured using a specialized beam-splitter rig to get perfectly aligned low-light and normal-light pairs.

Key Insights and Ideas

  • Unification is Better than Cascading: The "aha moment" here is that denoising, brightening, and upsampling aren't just related—they are mathematically compatible. By representing them as a single parametric filter, the model avoids the "error accumulation" common in multi-stage pipelines.
  • Temporal Context is King: Most illumination models work frame-by-frame. By using a 3D filter, this model uses information from neighboring frames to decide how to brighten a pixel, leading to much more stable video with less flickering.
  • Efficiency through Representation: Because the network predicts filter parameters rather than raw pixels for three different tasks, it achieves state-of-the-art results with a significantly lower computational footprint than cascading SOTA models.

Key Results

  • Quantitative Superiority: DP3DF consistently outperformed top-tier SR models (like BasicVSR++ and EDVR) across all datasets. For example, on the SDSD Indoor dataset, it achieved a PSNR of 27.11, significantly higher than the 21.02 of BasicVSR++.
  • Speed: The model is remarkably fast. In benchmarks, it processed 960×512 images in a fraction of the time required by other high-performance models like RBPN or Zooming Slow-Mo.
  • Human Preference: A user study with 80 participants showed that people overwhelmingly preferred the DP3DF results for their natural brightness, distinct contrast, and lack of "aliasing" (jagged edges).

Limitations and Future Work

While the results are impressive, the paper leaves a few stones unturned:

  • Fixed Scale Factor: The experiments focused on a 4x upsampling rate; it's unclear how the parametric filters scale to arbitrary or much higher resolutions.
  • Temporal Window: The model uses a relatively small temporal window ($T=3$ or $T=5$). Future work could explore integrating long-range temporal dependencies (like those found in Transformers) to handle even more complex motion.
  • Extreme Darkness: While it handles low light well, the "See-In-The-Dark" scenarios (near-total darkness) still present a massive challenge for any reconstruction-based approach.

Summary updated

Abstract

Despite the quality improvement brought by the recent methods, video super-resolution (SR) is still very challenging, especially for videos that are low-light and noisy. The current best solution is to subsequently employ best models of video SR, denoising, and illumination enhancement, but doing so often lowers the image quality, due to the inconsistency between the models. This paper presents a new parametric representation called the Deep Parametric 3D Filters (DP3DF), which incorporates local spatiotemporal information to enable simultaneous denoising, illumination enhancement, and SR efficiently in a single encoder-and-decoder network. Also, a dynamic residual frame is jointly learned with the DP3DF via a shared backbone to further boost the SR quality. We performed extensive experiments, including a large-scale user study, to show our method's effectiveness. Our method consistently surpasses the best state-of-the-art methods on all the challenging real datasets with top PSNR and user ratings, yet having a very fast run time.
BibTeX
@inproceedings{xu2023deep,
  author = {Xiaogang Xu and Ruixing Wang and Chi-Wing Fu and Jiaya Jia},
  title = {{Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25409},
}
Figure