High-Frequency Refinement for Sharper Video Super-Resolution

Vikram Singh, Akshay Sharma, Sudharshann Devanathan, Anurag MittalWACV 2020

TL;DR

This paper introduces HFR-Net, a novel video super-resolution network that explicitly refines high-frequency details to produce sharper output. It employs a unique "2-phase progressive-retrogressive training" method and a "dual motion warping" technique for improved motion handling, achieving state-of-the-art results.

What Problem Does This Paper Address?

Upscaling low-resolution videos to higher resolutions (video super-resolution) is a crucial task for improving visual quality, especially with the proliferation of high-resolution displays and low-quality video content. While existing methods can increase resolution, they often struggle to generate truly "sharp" images, meaning they fail to accurately predict and reconstruct fine details like object edges. This lack of sharpness is a significant limitation, as it detracts from the perceived quality and realism of the upscaled video. The paper aims to bridge this gap by focusing on the precise prediction and refinement of these high-frequency details.

Methods

The core of the paper is the HFR-Net (High-Frequency Refinement Network), a five-stage architecture designed to tackle the sharpness problem.

  • Dual Motion Warping (Stage 1): To better handle videos with varying motion intensities (both slow and fast), the authors propose a novel "dual motion warping" technique. This combines "direct warping" (where source frames are directly warped to the target frame) and "sequential warping" (where frames are warped iteratively). This hybrid approach aims to mitigate the weaknesses of each individual method when dealing with diverse motion patterns.
  • Feature Refinement (Stage 2): This stage extracts and refines features from the motion-compensated frames, preparing them for the upsampling process. It utilizes bidirectional multi-scale convolutional LSTMs to leverage temporal information.
  • Upsampling (Stage 3): This stage performs the initial upsampling of the target frame, generating an intermediate version. It employs 3D convolutions for temporal feature enrichment and 2D convolutions for channel feature improvement, along with dense connections for efficient training.
  • High-Frequency Refinement (Stage 4): This is a key innovation. The intermediate upsampled frame is decomposed into high-frequency (HF) and low-frequency (LF) components. The HF component is then explicitly refined using a convolutional network.
  • Frame Fusion (Stage 5): The refined HF frame and the LF frame are fused to produce the final, sharper upsampled output. This stage is trained using the novel "2-phase progressive-retrogressive training" method.

2-Phase Progressive-Retrogressive Training: This is a unique training paradigm.

  • Progressive Phase: The network is trained with the ground-truth frame as an input, allowing it to reach a very good state and produce high-quality predictions.
  • Retrogressive Phase: The ground-truth input is gradually replaced by a substitute generated by earlier stages of the network. This phase retrains the network to perform well without direct access to the ground-truth during inference, effectively making the network self-sufficient. This is crucial because ground-truth is not available during real-world inference.

Datasets

The paper uses publicly available datasets for training and evaluation:

  • Training and Validation: Vid4 (by Liu et al. [14]), Derf4 (by Wang et al. [22]), and SPMCS (by Tao et al. [20]).
  • Evaluation: The same datasets (Vid4, SPMCS, Derf4) are used for quantitative and qualitative comparisons.

No new benchmarks were introduced.

Key Insights and Ideas

  • Sharpness is Key: The paper strongly emphasizes that true video super-resolution quality hinges on the accurate prediction of high-frequency details, not just increasing pixel count.
  • Decomposition for Refinement: Explicitly separating high-frequency details and refining them independently before fusion is a powerful strategy for enhancing sharpness.
  • Dual Motion Warping for Robustness: Recognizing that a single warping strategy might fail on diverse motion types, the dual approach offers a more robust preprocessing step.
  • 2-Phase Training for Inference Independence: The "2-phase progressive-retrogressive training" is a clever way to leverage ground-truth during training for optimal performance while ensuring the model can operate effectively without it during inference. This is a significant contribution to training robust super-resolution models.
  • Stage-wise Contribution: Ablation studies reveal that each stage of HFR-Net contributes meaningfully, but the 2-phase trained fusion stage is identified as a major driver of performance.

Key Results

  • Quantitative Performance: HFR-Net outperforms state-of-the-art methods by a significant margin on Vid4, SPMCS, and Derf4 datasets in terms of PSNR and SSIM. For example, on the Vid4 dataset, HFR-Net achieves a PSNR of 29.30 and SSIM of 88.49, compared to the next best method's PSNR of 27.37 and SSIM of 83.34.
  • Qualitative Performance: Visualizations (Figure 8) demonstrate that HFR-Net produces noticeably sharper and more visually appealing upsampled frames compared to other methods, closely resembling the ground-truth.
  • Dual Motion Warping Effectiveness: Table 2 shows that dual motion warping consistently outperforms no warping, direct warping, and sequential warping across all tested datasets.
  • 2-Phase Training Effectiveness: Figure 9 and Table 3 illustrate that the 2-phase training method leads to better PSNR values than conventional 1-phase training, both during training and inference, indicating it sets the network in a better local-minima.

Limitations and Future Work

  • Training Time: The paper acknowledges that the 2-phase training is slower than 1-phase training, taking approximately double the time. However, they argue that inference time is unaffected, making it a worthwhile trade-off for performance.
  • Future Work: The authors plan to apply their proposed ideas (HFR-Net architecture, 2-phase training, and dual motion warping) to other image and video restoration tasks such as image super-resolution, deraining, dehazing, and deblurring. Preliminary results on these tasks are promising.

Summary updated

BibTeX
@inproceedings{singh2020high,
  author = {Vikram Singh and Akshay Sharma and Sudharshann Devanathan and Anurag Mittal},
  title = {{High-Frequency Refinement for Sharper Video Super-Resolution}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure