Abstract
Reference-based Video Super-Resolution (RefVSR) aims to enhance low-resolution (LR) video sequences by leveraging textures from a high-resolution (HR) reference image. However, existing methods often struggle with the resolution gap between HR and LR inputs and the accumulation of temporal errors. We present EFENet (Enhanced Flow Estimation Network), which introduces a Flow Refinement Module that utilizes global temporal information to correct alignment errors. By treating flow refinement as a global optimization problem across the sequence, EFENet significantly improves clarity and handles complex non-rigid motions. Our method achieves state-of-the-art performance on benchmarks like Vimeo90K and MPII.
Introduction
Super-resolving video is a challenging task. Traditional Video Super-Resolution (VSR) relies on neighboring LR frames, which are inherently limited in detail. Reference-based Super-Resolution (RefSR) attempts to circumvent this by "borrowing" high-frequency textures from a separate HR reference image. Applying RefSR to video (RefVSR) introduces two primary challenges: the resolution gap, making pixel matching difficult between sharp HR and blurry LR frames, and temporal drift, where independent frame alignment leads to inconsistent results or recursive errors.
Methodology
EFENet addresses these issues through a three-stage pipeline:
- Shared Flow Estimator: Estimates pixel movement (flow) between the HR reference and each LR frame in the sequence independently.
- Flow Refinement Module: A U-Net-style architecture that processes the entire stack of estimated flows. By leveraging global temporal context, it smooths motion estimates and corrects errors that local, frame-by-frame methods might miss.
- Encoder-Decoder Synthesis: The refined flow aligns HR reference features with the LR frame. These features are then fused with LR features through a deep decoder to generate the final HR output.
The core insight is that global optimization over multiple frames prevents the "chain" of errors typical in recursive warping methods. This approach is particularly effective for non-rigid motions, such as human movement, where standard rigid alignment fails.
Experiments and Results
EFENet was evaluated on the Vimeo90K and MPII datasets using a 4x upscaling factor.
- Quantitative Results: On the Vimeo90K dataset, EFENet achieved a PSNR of 39.25 dB, significantly outperforming the previous leader, CrossNet++ (38.95 dB), and nearly 4 dB higher than standard VSR methods like RBPN.
- Qualitative Results: Visual comparisons show that EFENet produces much sharper textures and fewer "ghosting" artifacts in areas with fast motion compared to previous methods. It successfully transfers fine details from the reference image even when the viewpoint has shifted significantly.
Conclusion
EFENet represents a significant advancement in RefVSR by prioritizing global flow refinement. While the computational overhead of global processing and the dependency on reference overlap are areas for future optimization, the current results set a new benchmark for video clarity and temporal consistency.
Summary updated