Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations

Peng Yi, Zhongyuan Wang, Kui Jiang, Junjun Jiang, Jiayi Ma

Abstract

Video Super-Resolution (VSR) aims to reconstruct high-resolution (HR) frames from low-resolution (LR) sequences by leveraging temporal information. Traditional VSR methods often rely on Motion Estimation and Motion Compensation (ME&MC) for frame alignment, which introduces significant computational overhead and potential artifacts. We present the Progressive Fusion Video Super-Resolution Network (PFNL), an alignment-free architecture that employs a Progressive Fusion Residual Block (PFRB) and an improved Non-Local Residual Block (NLRB). PFNL achieves a 0.96 dB PSNR improvement over previous state-of-the-art methods on standard benchmarks while running three times faster and utilizing approximately half the parameters.

Introduction

The primary challenge in Video Super-Resolution is effective temporal fusion—the process of combining information from multiple neighboring frames to enhance the current reference frame. Existing approaches typically suffer from two main issues:

  1. Inefficient Fusion Strategies: Methods like direct concatenation or 3D convolutions often fail to capture complex inter-frame correlations or become computationally prohibitive.
  2. The ME&MC Burden: Reliance on explicit motion vectors for alignment adds complexity, complicates training, and can lead to "flickering" artifacts if estimation is inaccurate.

PFNL addresses these challenges by replacing explicit alignment with a more robust, attention-based mechanism and a progressive fusion strategy.

Methods

The PFNL architecture is built upon two key innovations designed to exploit spatio-temporal correlations efficiently.

Progressive Fusion Residual Block (PFRB)

Rather than fusing all temporal information in a single step, the PFRB adopts a multi-channel, iterative approach:

  • Extraction: Spatial features are extracted independently from each input frame.
  • Aggregation & Distillation: These features are concatenated and passed through a $1 \times 1$ convolution layer. This "bottleneck" distills the most relevant temporal information relative to the reference frame.
  • Refinement: The distilled features are merged back with the original spatial features, allowing the network to progressively integrate temporal context while preserving spatial detail.

Improved Non-Local Residual Block (NLRB)

To capture long-range dependencies across the spatio-temporal volume, PFNL utilizes non-local operations. Standard non-local blocks are often too memory-intensive for high-resolution video. PFNL introduces a down-scaling factor ($r$) within the block, performing heavy matrix multiplications in a lower-dimensional space before upscaling. This optimization makes non-local attention feasible for full HD video processing on standard hardware.

Comparison of Non-local operation versus traditional Motion Compensation for temporal alignment.

By using non-local operations instead of motion compensation, the model becomes more robust to complex, non-linear motion and avoids the artifacts associated with poor alignment.

Results and Discussion

PFNL was evaluated on standard benchmarks like Vid4 and a collection of high-resolution sequences ($1272 \times 720$).

Performance Metrics

  • Accuracy: PFNL outperformed the previous state-of-the-art (DUF_52L) by an average of 0.96 dB PSNR.
  • Efficiency: The model is approximately 3x faster than DUF_52L, processing a 1080p frame in ~741ms compared to 2754ms.
  • Model Size: PFNL is highly compact, requiring only 3.003M parameters, nearly half of the 5.829M required by DUF_52L.

Visual Quality

Qualitative analysis demonstrates that PFNL excels at reconstructing fine textures and high-frequency details, such as text and complex patterns, which are often blurred or distorted by alignment-based methods.

Conclusion

The Progressive Fusion Video Super-Resolution Network demonstrates that attention-based non-local operations and progressive fusion are superior to traditional motion compensation for VSR. By distilling temporal information through efficient bottlenecks and leveraging down-scaled non-local blocks, PFNL provides a high-performance, lightweight solution for modern video enhancement tasks.

Summary updated

Abstract

Most previous fusion strategies either fail to fully utilize temporal information or cost too much time, and how to effectively fuse temporal information from consecutive frames plays an important role in video super-resolution (SR). In this study, we propose a novel progressive fusion network for video SR, which is designed to make better use of spatio-temporal information and is proved to be more efficient and effective than the existing direct fusion, slow fusion or 3D convolution strategies. Under this progressive fusion framework, we further introduce an improved non-local operation to avoid the complex motion estimation and motion compensation (ME&MC) procedures as in previous video SR approaches. Extensive experiments on public datasets demonstrate that our method surpasses state-of-the-art with 0.96 dB in average, and runs about 3 times faster, while requires only about half of the parameters.
BibTeX
@inproceedings{yi2019progressive,
  author = {Peng Yi and Zhongyuan Wang and Kui Jiang and Junjun Jiang and Jiayi Ma},
  title = {{Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00320},
}
Figure