Abstract
Video Super-Resolution (VSR) models trained on synthetic datasets often fail when applied to real-world videos characterized by complex sensor noise, motion blur, and compression artifacts. This paper addresses two primary bottlenecks in real-world VSR: the scarcity of paired $4\times$ datasets and the instability of inter-frame alignment. We introduce MVSR4×, a large-scale real-world dataset captured with a dual-camera smartphone setup providing a natural $4\times$ scale factor. Furthermore, we propose EAVSR, a framework featuring a Multi-layer Adaptive Spatial Transform Network (MultiAdaSTN) that synergizes the stability of optical flow with the flexibility of deformable convolutions. EAVSR achieves state-of-the-art results on both $2\times$ and $4\times$ real-world benchmarks.
Introduction
Most existing VSR models are optimized for synthetic degradations, such as bicubic downsampling. However, real-world video presents a "wild" environment where standard alignment techniques—like optical flow or Deformable Convolutional Networks (DCN)—frequently struggle. Optical flow is stable but often too rigid to capture fine details, while DCN is flexible but prone to instability during training on noisy data. This paper bridges the gap between synthetic and real-world performance through a new benchmark dataset and a hybrid alignment framework.
The MVSR4× Dataset
To facilitate research into high-magnification real-world VSR, we present MVSR4×. Unlike previous datasets like RealVSR which focus on $2\times$ upscaling, MVSR4× provides a challenging $4\times$ scale factor.
The dataset was captured using a Huawei P50 Pro smartphone's dual-view function, recording simultaneously with a 13mm ultra-wide lens (LR) and a 90mm telephoto lens (HR). It comprises 300 high-quality sequences of 100 frames each, covering a wide range of indoor and outdoor scenes, lighting conditions, and motion speeds. A robust pre-processing pipeline involving guided filtering and PWC-Net was employed to align the HR ground truth with the LR input, mitigating parallax and color mismatches.
The EAVSR Framework
The core of the proposed EAVSR (Effective Alignment for VSR) is the MultiAdaSTN module. This module avoids the binary choice between flow and DCN by adopting a hierarchical, hybrid approach.
ResflowNet and DeformNet
MultiAdaSTN operates in two primary stages:
- ResflowNet: This component uses a pre-trained optical flow network (e.g., SPyNet) as a baseline and refines it through a coarse-to-fine pyramid. By learning the "residual flow" rather than the entire displacement field, the model achieves significantly higher stability on real-world textures.
- DeformNet: For final pixel-level alignment, EAVSR utilizes a modified AdaSTN v2. Unlike the standard AdaSTN, version 2 allows for channel-specific flexibility and incorporates a masking mechanism to suppress unreliable or out-of-bounds information.
The framework is further enhanced by bidirectional propagation (EAVSR) or second-order grid propagation (EAVSR+) to aggregate temporal information across the sequence.
Experimental Results
EAVSR+ demonstrates superior performance across major real-world benchmarks. On the RealVSR ($2\times$) dataset, it achieved a PSNR of 24.41 dB, surpassing BasicVSR++ by 0.17 dB. On the more challenging MVSR4× ($4\times$) dataset, EAVSR+ reached 23.94 dB, outperforming competitive models like ETDM and TTVSR.
Qualitatively, EAVSR+ restores sharper edges and finer structural details, such as building contours and text, which are often blurred by other methods. For applications requiring high perceptual quality, the GAN-based variant, EAVSRGAN+, produces the most photo-realistic results with minimal artifacts.
Conclusion
This work highlights the critical importance of real-world training data and robust inter-frame alignment. While MVSR4× and EAVSR significantly advance the state-of-the-art, challenges remain regarding hardware-induced parallax and the computational overhead of multi-layer refinement. Future work will focus on optimizing these models for real-time mobile deployment.
Summary updated