EFENet: Reference-Based Video Super-Resolution with Enhanced Flow Estimation

Yaping Zhao, Mengqi Ji, Ruqi Huang, Bin Wang, Shengjin Wang

Abstract

Reference-based Video Super-Resolution (RefVSR) aims to enhance low-resolution (LR) video sequences by leveraging textures from a high-resolution (HR) reference image. However, existing methods often struggle with the resolution gap between HR and LR inputs and the accumulation of temporal errors. We present EFENet (Enhanced Flow Estimation Network), which introduces a Flow Refinement Module that utilizes global temporal information to correct alignment errors. By treating flow refinement as a global optimization problem across the sequence, EFENet significantly improves clarity and handles complex non-rigid motions. Our method achieves state-of-the-art performance on benchmarks like Vimeo90K and MPII.

Introduction

Super-resolving video is a challenging task. Traditional Video Super-Resolution (VSR) relies on neighboring LR frames, which are inherently limited in detail. Reference-based Super-Resolution (RefSR) attempts to circumvent this by "borrowing" high-frequency textures from a separate HR reference image. Applying RefSR to video (RefVSR) introduces two primary challenges: the resolution gap, making pixel matching difficult between sharp HR and blurry LR frames, and temporal drift, where independent frame alignment leads to inconsistent results or recursive errors.

Overview of the Reference-Based Video Super-resolution framework, showing the integration of HR reference and LR video frames.

Methodology

EFENet addresses these issues through a three-stage pipeline:

  1. Shared Flow Estimator: Estimates pixel movement (flow) between the HR reference and each LR frame in the sequence independently.
  2. Flow Refinement Module: A U-Net-style architecture that processes the entire stack of estimated flows. By leveraging global temporal context, it smooths motion estimates and corrects errors that local, frame-by-frame methods might miss.
  3. Encoder-Decoder Synthesis: The refined flow aligns HR reference features with the LR frame. These features are then fused with LR features through a deep decoder to generate the final HR output.
Comparison of different flow estimation strategies in video super-resolution, highlighting the global approach (d) used in EFENet.

The core insight is that global optimization over multiple frames prevents the "chain" of errors typical in recursive warping methods. This approach is particularly effective for non-rigid motions, such as human movement, where standard rigid alignment fails.

Experiments and Results

EFENet was evaluated on the Vimeo90K and MPII datasets using a 4x upscaling factor.

  • Quantitative Results: On the Vimeo90K dataset, EFENet achieved a PSNR of 39.25 dB, significantly outperforming the previous leader, CrossNet++ (38.95 dB), and nearly 4 dB higher than standard VSR methods like RBPN.
  • Qualitative Results: Visual comparisons show that EFENet produces much sharper textures and fewer "ghosting" artifacts in areas with fast motion compared to previous methods. It successfully transfers fine details from the reference image even when the viewpoint has shifted significantly.

Conclusion

EFENet represents a significant advancement in RefVSR by prioritizing global flow refinement. While the computational overhead of global processing and the dependency on reference overlap are areas for future optimization, the current results set a new benchmark for video clarity and temporal consistency.

Summary updated

Abstract

In this paper, we consider the problem of reference-based video super-resolution(RefVSR), i.e., how to utilize a high-resolution (HR) reference frame to super-resolve a low-resolution (LR) video sequence. The existing approaches to RefVSR essentially attempt to align the reference and the input sequence, in the presence of resolution gap and long temporal range. However, they either ignore temporal structure within the input sequence, or suffer accumulative alignment errors. To address these issues, we propose EFENet to exploit simultaneously the visual cues contained in the HR reference and the temporal information contained in the LR sequence. EFENet first globally estimates cross-scale flow between the reference and each LR frame. Then our novel flow refinement module of EFENet refines the flow regarding the furthest frame using all the estimated flows, which leverages the global temporal information within the sequence and therefore effectively reduces the alignment errors. We provide comprehensive evaluations to validate the strengths of our approach, and to demonstrate that the proposed framework outperforms the state-of-the-art methods.
BibTeX
@inproceedings{zhao2021efenet,
  author = {Yaping Zhao and Mengqi Ji and Ruqi Huang and Bin Wang and Shengjin Wang},
  title = {{EFENet: Reference-Based Video Super-Resolution with Enhanced Flow Estimation}},
  booktitle = {CICAI},
  year = {2021},
  url = {https://openreview.net/forum?id=X9-lH3Uv40},
}
Figure