Joint Video Rolling Shutter Correction and Super-Resolution

Akash Gupta, Sudhir Kumar Singh, Amit K. Roy-ChowdhuryWACV 2023 · OpenReview

TL;DR

This paper introduces PatchNet, a novel deep learning architecture that jointly addresses rolling shutter (RS) artifact correction and super-resolution in videos. By leveraging patch-level attention and deformable convolutions, PatchNet effectively reconstructs high-resolution, global shutter video from low-resolution RS footage, outperforming existing methods that tackle these tasks separately.

What Problem Does This Paper Address?

Modern CMOS cameras, prevalent in mobile devices, often suffer from rolling shutter (RS) artifacts. These distortions, caused by the sequential scanning of pixels, manifest as skew, wobble, or smear when the camera or scene is in motion. While existing super-resolution (SR) techniques assume global shutter (GS) video and ignore RS artifacts, and RS correction methods often don't address SR, this paper tackles the more realistic and challenging problem of jointly correcting RS artifacts and enhancing resolution in videos. This is crucial for applications requiring high-quality, high-resolution video from ubiquitous RS cameras.

Methods

The core of the proposed solution is the Patch Attention Network (PatchNet). This architecture is designed to learn both local pixel-level details and global patch-level relationships.

  • Deformable Convolution: Used to capture local pixel-level information and geometric variations within the RS frames.
  • Patch-Level Attention: This is the key novelty. PatchNet leverages the "patch recurrence property," the observation that small image patches tend to repeat across frames. It uses an attention mechanism to exploit information from neighboring patches in the feature space.
  • Bi-directional Motion Fields: The model utilizes optical flow (forward and backward) to guide the attention mechanism, helping it find correlated information across patches and time.
  • Joint Optimization: Unlike prior work that separates RS correction and SR, PatchNet jointly optimizes for both tasks in the feature space. This allows it to learn a more comprehensive representation that benefits both correction and enhancement.
  • Feature Space Processing: The SR is performed in the feature space before generating the final high-resolution global shutter frames, which is more efficient and effective than cascaded approaches.

Datasets

The paper uses two datasets for training and evaluation:

  • BS-RSCD Dataset: A dynamic urban environment dataset containing both ego-motion and object-motion. It includes 80 short video sequences with corresponding RS and GS frames. The authors down-sample the RS frames to 320x240 for training.
  • Fastec-RS Dataset: A synthetic dataset captured using a professional high-speed global shutter camera. It synthesizes RS images by sequentially copying rows from GS frames and then down-sampling them to 320x240 for training.

Key Insights and Ideas

  • Jointly tackling RS and SR is crucial: Separating these tasks limits performance because they are inherently intertwined. RS correction provides a better base for SR, and SR can help refine the corrected details.
  • Patch recurrence is a powerful signal: Exploiting the fact that image patches repeat across frames, especially in video, allows the model to infer missing information and improve reconstruction.
  • Attention in feature space is key: Applying attention at the patch level in the latent feature space, guided by motion, enables the model to effectively aggregate information from relevant neighboring patches.
  • Deformable convolutions complement patch attention: While patch attention captures global context, deformable convolutions handle local geometric distortions and fine details, creating a robust pipeline.

Key Results

  • Quantitative Performance: PatchNet achieves state-of-the-art results on both BS-RSCD and Fastec-RS datasets. On BS-RSCD, it shows a 2.32dB improvement in average PSNR compared to the best combination of existing RS correction (JCD) and SR (EDSR) methods. It also outperforms methods that only perform RS correction on high-resolution inputs.
  • Qualitative Performance: Visual results demonstrate that PatchNet produces significantly sharper and more detailed frames with fewer artifacts compared to baseline methods. The paper highlights that PatchNet's joint learning approach leads to visually superior results, especially in text regions.
  • Patch Size Impact: Ablation studies show that an 8x8 patch size for attention yields the best performance, indicating a good balance between capturing local details and global context.

Limitations and Future Work

  • Camera Configuration Dependency: The current model is trained on specific camera configurations and may not generalize perfectly to unseen settings.
  • Future Directions: The authors suggest developing models that are agnostic to camera configurations by incorporating specific parameters like readout and exposure times. This would likely require semi-supervised or self-supervised learning approaches due to the difficulty of obtaining such data.

Summary updated

Abstract

With the prevalence of CMOS cameras in many computer vision applications, there is an increase in the appearance of rolling shutter (RS) artifacts in captured videos. However, existing video super-resolution algorithms assume that the motion is globally consistent in each video frame and no rolling shutter effect is present. The problem of video super-resolution for video captured using RS cameras is challenging as the model needs to learn the row-wise local pixel displacements and the global structure of the frame for RS correction and super-resolution, simultaneously. Different from existing works, we address a more realistic problem of joint rolling shutter correction and super-resolution (RS-SR). We introduce a novel architecture, deformable Patch Attention Network (PatchNet), that utilizes patch-recurrence property along with deformable receptive fields to learn the global and local structure of the video. Specifically, PatchNet leverages bi-directional motion field in the feature space to extract relevant information from neighboring patches using attention mechanism, and deformable fields using deformable convolutions to extract local pixel-level information for joint rolling shutter correction and super-resolution. Our work is the first to tackle the task of RS correction and super-resolution on the recently released BS-RSCD dataset. Experiments on the BS-RSCD and FastecRS datasets demonstrate that our model performs favorably against various state-of-the-art approaches. Project details are available at https://akashagupta.com/publication/wacv23_patchnet/project.html
BibTeX
@inproceedings{gupta2023joint,
  author = {Akash Gupta and Sudhir Kumar Singh and Amit K. Roy-Chowdhury},
  title = {{Joint Video Rolling Shutter Correction and Super-Resolution}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=ZWGESFI36EY},
}
Figure