Real-World Video for Zoom Enhancement based on Spatio-Temporal Coupling

Zhiling Guo, Yinqiang Zheng, Haoran Zhang, Xiaodan Shi, Zekun Cai, Ryosuke Shibasaki, Jinyue YanarXiv 2023 · DOI · arXiv

Introduction

Digital zoom is a ubiquitous feature in modern photography, yet its quality remains a significant bottleneck. Traditional methods rely on simple cropping and upsampling, which frequently introduce noise, artifacts, and a loss of fine detail. While Super-Resolution (SR) techniques using artificial intelligence have shown promise, they are often trained on simulated datasets where low-resolution (LR) images are merely downsampled versions of high-resolution (HR) counterparts.

In real-world scenarios, the relationship between wide-angle and zoomed-in shots is complicated by lens distortion, sensor noise, and physical camera misalignment. This research addresses these challenges by introducing a novel dataset and a spatio-temporal training framework specifically designed for real-world video zoom enhancement.

The VideoRAW Dataset: Bridging the Reality Gap

To move beyond simulated data, the researchers developed VideoRAW, the first large-scale dataset of paired real-world LR and HR videos. Capturing such data requires a specialized hardware setup to ensure ground-truth accuracy. The team utilized a co-axial optical system featuring a beam splitter, which allows two cameras with different focal lengths to share the same optical axis.

VideoRAW consists of 84 video pairs captured across diverse urban environments. Each clip contains 200 frames at a resolution of 1384 × 1032. Crucially, the dataset provides 16-bit RAW data for the LR input. By using RAW data, the model avoids the artifacts typically introduced by in-camera image signal processors (ISPs), such as demosaicing and sharpening, allowing the AI to work with the purest sensor information.

Spatio-Temporal Coupling Loss (STCL)

Training on real-world data presents a significant alignment challenge, as LR and HR frames are rarely perfectly synchronized at the pixel level due to mechanical offsets. To overcome this, the authors proposed the Spatio-Temporal Coupling Loss (STCL). This framework employs two key strategies:

  1. Spatial Alignment: Instead of enforcing strict pixel-to-pixel matching, the model uses a contextual loss to align feature distributions. This makes the training robust to the small physical misalignments inherent in dual-camera systems.
  2. Temporal Fusion: The framework leverages temporal consistency by weighting adjacent frames. This allows the model to "borrow" high-frequency details from neighboring frames to compensate for noise or blur in any single frame, leading to more stable and detailed video output.

Key Results and Discussion

The proposed method demonstrates superior performance over existing baselines in both quantitative and qualitative evaluations. In benchmark tests, the STCL-based approach achieved a Peak Signal-to-Noise Ratio (PSNR) of 30.21, significantly outperforming standard $L_2$ loss (29.40) and spatial-only methods like CoBi (29.47).

Visual comparison of zoom enhancement methods on the VideoRAW dataset, showing the superiority of the proposed spatio-temporal coupling approach (E) against standard upsampling and spatial-only methods.

Visually, the method produces significantly sharper results, particularly in challenging areas like text on license plates and fine vegetation textures where other methods produce "mushy" artifacts. Human preference studies further validate these findings, with participants preferring the STCL results in over 50% of pairwise comparisons. Furthermore, the STCL framework is architecture-agnostic and can be integrated into various existing SR models to enhance their performance on real-world video data.

Conclusion

This work represents a significant step forward in real-world video super-resolution. By providing the VideoRAW dataset and the STCL framework, the researchers have established a robust foundation for future developments in digital zoom technology. While current limitations include a fixed 4x zoom ratio and a 15fps capture rate, the framework's ability to fuse spatio-temporal features provides a clear path toward high-fidelity digital zoom in consumer electronics.

Summary updated

Abstract

In recent years, single-frame image super-resolution (SR) has become more realistic by considering the zooming effect and using real-world short- and long-focus image pairs. In this paper, we further investigate the feasibility of applying realistic multi-frame clips to enhance zoom quality via spatio-temporal information coupling. Specifically, we first built a real-world video benchmark, VideoRAW, by a synchronized co-axis optical system. The dataset contains paired short-focus raw and long-focus sRGB videos of different dynamic scenes. Based on VideoRAW, we then presented a Spatio-Temporal Coupling Loss, termed as STCL. The proposed STCL is intended for better utilization of information from paired and adjacent frames to align and fuse features both temporally and spatially at the feature level. The outperformed experimental results obtained in different zoom scenarios demonstrate the superiority of integrating real-world video dataset and STCL into existing SR models for zoom quality enhancement, and reveal that the proposed method can serve as an advanced and viable tool for video zoom.
BibTeX
@misc{guo2023real,
  author = {Zhiling Guo and Yinqiang Zheng and Haoran Zhang and Xiaodan Shi and Zekun Cai and Ryosuke Shibasaki and Jinyue Yan},
  title = {{Real-World Video for Zoom Enhancement based on Spatio-Temporal Coupling}},
  year = {2023},
  eprint = {2306.13875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.13875},
}
Figure