Benchmark Dataset and Effective Inter-Frame Alignment for Real-World Video Super-Resolution

Ruohao Wang, Xiaohui Liu, Zhilu Zhang, Xiaohe Wu, Chun-Mei Feng, Lei Zhang, Wangmeng Zuo

Abstract

Video Super-Resolution (VSR) models trained on synthetic datasets often fail when applied to real-world videos characterized by complex sensor noise, motion blur, and compression artifacts. This paper addresses two primary bottlenecks in real-world VSR: the scarcity of paired $4\times$ datasets and the instability of inter-frame alignment. We introduce MVSR4×, a large-scale real-world dataset captured with a dual-camera smartphone setup providing a natural $4\times$ scale factor. Furthermore, we propose EAVSR, a framework featuring a Multi-layer Adaptive Spatial Transform Network (MultiAdaSTN) that synergizes the stability of optical flow with the flexibility of deformable convolutions. EAVSR achieves state-of-the-art results on both $2\times$ and $4\times$ real-world benchmarks.

Introduction

Most existing VSR models are optimized for synthetic degradations, such as bicubic downsampling. However, real-world video presents a "wild" environment where standard alignment techniques—like optical flow or Deformable Convolutional Networks (DCN)—frequently struggle. Optical flow is stable but often too rigid to capture fine details, while DCN is flexible but prone to instability during training on noisy data. This paper bridges the gap between synthetic and real-world performance through a new benchmark dataset and a hybrid alignment framework.

The MVSR4× Dataset

To facilitate research into high-magnification real-world VSR, we present MVSR4×. Unlike previous datasets like RealVSR which focus on $2\times$ upscaling, MVSR4× provides a challenging $4\times$ scale factor.

The MVSR4x data collection pipeline utilizes dual cameras (13mm and 90mm) to capture paired LR and HR video sequences.

The dataset was captured using a Huawei P50 Pro smartphone's dual-view function, recording simultaneously with a 13mm ultra-wide lens (LR) and a 90mm telephoto lens (HR). It comprises 300 high-quality sequences of 100 frames each, covering a wide range of indoor and outdoor scenes, lighting conditions, and motion speeds. A robust pre-processing pipeline involving guided filtering and PWC-Net was employed to align the HR ground truth with the LR input, mitigating parallax and color mismatches.

The EAVSR Framework

The core of the proposed EAVSR (Effective Alignment for VSR) is the MultiAdaSTN module. This module avoids the binary choice between flow and DCN by adopting a hierarchical, hybrid approach.

The MultiAdaSTN architecture consists of ResflowNet for coarse-to-fine flow refinement and DeformNet for pixel-perfect alignment using AdaSTN v2.

ResflowNet and DeformNet

MultiAdaSTN operates in two primary stages:

  1. ResflowNet: This component uses a pre-trained optical flow network (e.g., SPyNet) as a baseline and refines it through a coarse-to-fine pyramid. By learning the "residual flow" rather than the entire displacement field, the model achieves significantly higher stability on real-world textures.
  2. DeformNet: For final pixel-level alignment, EAVSR utilizes a modified AdaSTN v2. Unlike the standard AdaSTN, version 2 allows for channel-specific flexibility and incorporates a masking mechanism to suppress unreliable or out-of-bounds information.

The framework is further enhanced by bidirectional propagation (EAVSR) or second-order grid propagation (EAVSR+) to aggregate temporal information across the sequence.

Experimental Results

EAVSR+ demonstrates superior performance across major real-world benchmarks. On the RealVSR ($2\times$) dataset, it achieved a PSNR of 24.41 dB, surpassing BasicVSR++ by 0.17 dB. On the more challenging MVSR4× ($4\times$) dataset, EAVSR+ reached 23.94 dB, outperforming competitive models like ETDM and TTVSR.

Qualitatively, EAVSR+ restores sharper edges and finer structural details, such as building contours and text, which are often blurred by other methods. For applications requiring high perceptual quality, the GAN-based variant, EAVSRGAN+, produces the most photo-realistic results with minimal artifacts.

Conclusion

This work highlights the critical importance of real-world training data and robust inter-frame alignment. While MVSR4× and EAVSR significantly advance the state-of-the-art, challenges remain regarding hardware-induced parallax and the computational overhead of multi-layer refinement. Future work will focus on optimizing these models for real-time mobile deployment.

Summary updated

Abstract

Video super-resolution (VSR) aiming to reconstruct a high-resolution (HR) video from its low-resolution (LR) counterpart has made tremendous progress in recent years. However, it remains challenging to deploy existing VSR methods to real-world data with complex degradations. On the one hand, there are few well-aligned real-world VSR datasets, especially with large super-resolution scale factors, which limits the development of real-world VSR tasks. On the other hand, alignment algorithms in existing VSR methods perform poorly for real-world videos, leading to unsatisfactory results. As an attempt to address the aforementioned issues, we build a real-world ×4 VSR dataset, namely MVSR4×, where low- and high-resolution videos are captured with different focal length lenses of a smartphone, respectively. Moreover, we propose an effective alignment method for real-world VSR, namely EAVSR. EAVSR takes the proposed multi-layer adaptive spatial transform network (MultiAdaSTN) to refine the offsets provided by the pre-trained optical flow estimation network. Experimental results on RealVSR and MVSR4× datasets show the effectiveness and practicality of our method, and we achieve state-of-the-art performance in real-world VSR task. The dataset and code will be available at https://github.com/HITRainer/EAVSR.
BibTeX
@inproceedings{wang2022benchmark,
  author = {Ruohao Wang and Xiaohui Liu and Zhilu Zhang and Xiaohe Wu and Chun-Mei Feng and Lei Zhang and Wangmeng Zuo},
  title = {{Benchmark Dataset and Effective Inter-Frame Alignment for Real-World Video Super-Resolution}},
  booktitle = {Workshops @ CVPR},
  year = {2022},
}
Figure