Abstract
Video Super-Resolution (VSR) aims to reconstruct high-resolution (HR) frames from low-resolution (LR) sequences by leveraging temporal information. Traditional VSR methods often rely on Motion Estimation and Motion Compensation (ME&MC) for frame alignment, which introduces significant computational overhead and potential artifacts. We present the Progressive Fusion Video Super-Resolution Network (PFNL), an alignment-free architecture that employs a Progressive Fusion Residual Block (PFRB) and an improved Non-Local Residual Block (NLRB). PFNL achieves a 0.96 dB PSNR improvement over previous state-of-the-art methods on standard benchmarks while running three times faster and utilizing approximately half the parameters.
Introduction
The primary challenge in Video Super-Resolution is effective temporal fusion—the process of combining information from multiple neighboring frames to enhance the current reference frame. Existing approaches typically suffer from two main issues:
- Inefficient Fusion Strategies: Methods like direct concatenation or 3D convolutions often fail to capture complex inter-frame correlations or become computationally prohibitive.
- The ME&MC Burden: Reliance on explicit motion vectors for alignment adds complexity, complicates training, and can lead to "flickering" artifacts if estimation is inaccurate.
PFNL addresses these challenges by replacing explicit alignment with a more robust, attention-based mechanism and a progressive fusion strategy.
Methods
The PFNL architecture is built upon two key innovations designed to exploit spatio-temporal correlations efficiently.
Progressive Fusion Residual Block (PFRB)
Rather than fusing all temporal information in a single step, the PFRB adopts a multi-channel, iterative approach:
- Extraction: Spatial features are extracted independently from each input frame.
- Aggregation & Distillation: These features are concatenated and passed through a $1 \times 1$ convolution layer. This "bottleneck" distills the most relevant temporal information relative to the reference frame.
- Refinement: The distilled features are merged back with the original spatial features, allowing the network to progressively integrate temporal context while preserving spatial detail.
Improved Non-Local Residual Block (NLRB)
To capture long-range dependencies across the spatio-temporal volume, PFNL utilizes non-local operations. Standard non-local blocks are often too memory-intensive for high-resolution video. PFNL introduces a down-scaling factor ($r$) within the block, performing heavy matrix multiplications in a lower-dimensional space before upscaling. This optimization makes non-local attention feasible for full HD video processing on standard hardware.
By using non-local operations instead of motion compensation, the model becomes more robust to complex, non-linear motion and avoids the artifacts associated with poor alignment.
Results and Discussion
PFNL was evaluated on standard benchmarks like Vid4 and a collection of high-resolution sequences ($1272 \times 720$).
Performance Metrics
- Accuracy: PFNL outperformed the previous state-of-the-art (DUF_52L) by an average of 0.96 dB PSNR.
- Efficiency: The model is approximately 3x faster than DUF_52L, processing a 1080p frame in ~741ms compared to 2754ms.
- Model Size: PFNL is highly compact, requiring only 3.003M parameters, nearly half of the 5.829M required by DUF_52L.
Visual Quality
Qualitative analysis demonstrates that PFNL excels at reconstructing fine textures and high-frequency details, such as text and complex patterns, which are often blurred or distorted by alignment-based methods.
Conclusion
The Progressive Fusion Video Super-Resolution Network demonstrates that attention-based non-local operations and progressive fusion are superior to traditional motion compensation for VSR. By distilling temporal information through efficient bottlenecks and leveraging down-scaled non-local blocks, PFNL provides a high-performance, lightweight solution for modern video enhancement tasks.
Summary updated