Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution

Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, Jiaya Jia

Abstract

Real-world video restoration often requires addressing multiple degradations simultaneously, such as low resolution, high noise, and poor illumination. Traditional approaches rely on cascaded pipelines of specialized models, which are computationally expensive and prone to artifact amplification. This paper introduces Deep Parametric 3D Filters (DP3DF), a unified neural network architecture that jointly performs video super-resolution, denoising, and illumination enhancement. By predicting dynamic, pixel-specific 3D filters, the proposed method achieves state-of-the-art visual quality and high inference speeds.

1. Introduction

Capturing high-quality video in low-light conditions remains a significant challenge, often resulting in footage that is dark, noisy, and low-resolution (LLN). The industry standard for converting LLN videos to high-resolution, normal-light, noise-free (HNN) outputs typically involves chaining three separate neural networks: one for denoising, one for illumination enhancement, and one for super-resolution (SR).

However, this cascaded approach has significant drawbacks. First, running three distinct models is computationally intensive. Second, independent training leads to inconsistency; for instance, artifacts introduced during denoising are often amplified by the subsequent SR stage. We address these issues by proposing a single, end-to-end network capable of handling all three tasks simultaneously through a unified representation.

2. Deep Parametric 3D Filters (DP3DF)

The core of our approach is the Deep Parametric 3D Filter. Instead of directly regressing pixel values, the network predicts dynamic, pixel-specific 3D filters based on a local spatio-temporal window.

2.1 Unified Filter Representation

For each pixel, the network predicts a filter of size $k_h \times k_w \times k_t + 1$. The spatio-temporal components ($k_h \times k_w \times k_t$) perform convolution to align adjacent frames, suppress noise, and upscale resolution. The additional dimension acts as a pixel-wise illumination multiplier, inspired by Retinex theory, to brighten dark regions effectively.

2.2 Residual Learning and Optimization

To prevent over-smoothing—a common side effect of denoising filters—a parallel residual branch extracts high-frequency details from the encoder features. These details are integrated into the filtered frame for a sharper final output. The model is optimized end-to-end using L2 reconstruction losses and a spatial smoothness loss for the illumination map to ensure natural contrast.

3. Experimental Evaluation

We evaluate DP3DF on two real-world datasets:

  • SMID (Seeing Motion in the Dark): Static low-light videos converted to sRGB to test the LLN-to-HNN mapping.
  • SDSD (Seeing Dynamic Scene in the Dark): Challenging indoor and outdoor video pairs with moving subjects, testing temporal alignment and motion handling.

4. Results and Analysis

DP3DF demonstrates significant improvements over both standalone SOTA Video SR methods and complex cascaded pipelines.

Qualitative and quantitative comparison of DP3DF against baseline methods.

4.1 Quantitative Performance

On the SMID dataset, DP3DF achieved a PSNR of 25.73, outperforming the best cascaded baseline (25.22). On the dynamic SDSD Outdoor set, it reached 25.80 PSNR, compared to 25.01 for the cascaded approach.

4.2 Efficiency and Human Preference

The unified backbone allows DP3DF to process a 960x512 frame in under 0.05 seconds on a TITAN XP GPU. In a user study with 80 participants, DP3DF's outputs were overwhelmingly preferred for their natural brightness and lack of artifacts ($p < 0.001$).

4.3 Key Insights

  • Spatio-Temporal Synergy: Ablation studies show that the 3D nature of the filter is crucial; removing either spatial or temporal dimensions significantly degrades performance.
  • Artifact Mitigation: By mapping raw LLN input directly to HNN output, the model avoids the "telephone game" effect where errors from early stages are magnified later in the pipeline.

5. Conclusion

DP3DF provides a robust and efficient solution for joint video restoration tasks. While the current supervised approach relies on high-quality paired datasets, future work could explore self-supervised adaptations or dynamic filter scaling to further enhance performance in diverse real-world scenarios.

Summary updated

BibTeX
@inproceedings{xu2023deep,
  author = {Xiaogang Xu and Ruixing Wang and Chi-Wing Fu and Jiaya Jia},
  title = {{Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25409},
}
Figure