TL;DR
This paper introduces PatchNet, a novel deep learning architecture that jointly addresses rolling shutter (RS) artifact correction and super-resolution in videos. By leveraging patch-level attention and deformable convolutions, PatchNet effectively reconstructs high-resolution, global shutter video from low-resolution RS footage, outperforming existing methods that tackle these tasks separately.
What Problem Does This Paper Address?
Modern CMOS cameras, prevalent in mobile devices, often suffer from rolling shutter (RS) artifacts. These distortions, caused by the sequential scanning of pixels, manifest as skew, wobble, or smear when the camera or scene is in motion. While existing super-resolution (SR) techniques assume global shutter (GS) video and ignore RS artifacts, and RS correction methods often don't address SR, this paper tackles the more realistic and challenging problem of jointly correcting RS artifacts and enhancing resolution in videos. This is crucial for applications requiring high-quality, high-resolution video from ubiquitous RS cameras.
Methods
The core of the proposed solution is the Patch Attention Network (PatchNet). This architecture is designed to learn both local pixel-level details and global patch-level relationships.
- Deformable Convolution: Used to capture local pixel-level information and geometric variations within the RS frames.
- Patch-Level Attention: This is the key novelty. PatchNet leverages the "patch recurrence property," the observation that small image patches tend to repeat across frames. It uses an attention mechanism to exploit information from neighboring patches in the feature space.
- Bi-directional Motion Fields: The model utilizes optical flow (forward and backward) to guide the attention mechanism, helping it find correlated information across patches and time.
- Joint Optimization: Unlike prior work that separates RS correction and SR, PatchNet jointly optimizes for both tasks in the feature space. This allows it to learn a more comprehensive representation that benefits both correction and enhancement.
- Feature Space Processing: The SR is performed in the feature space before generating the final high-resolution global shutter frames, which is more efficient and effective than cascaded approaches.
Datasets
The paper uses two datasets for training and evaluation:
- BS-RSCD Dataset: A dynamic urban environment dataset containing both ego-motion and object-motion. It includes 80 short video sequences with corresponding RS and GS frames. The authors down-sample the RS frames to 320x240 for training.
- Fastec-RS Dataset: A synthetic dataset captured using a professional high-speed global shutter camera. It synthesizes RS images by sequentially copying rows from GS frames and then down-sampling them to 320x240 for training.
Key Insights and Ideas
- Jointly tackling RS and SR is crucial: Separating these tasks limits performance because they are inherently intertwined. RS correction provides a better base for SR, and SR can help refine the corrected details.
- Patch recurrence is a powerful signal: Exploiting the fact that image patches repeat across frames, especially in video, allows the model to infer missing information and improve reconstruction.
- Attention in feature space is key: Applying attention at the patch level in the latent feature space, guided by motion, enables the model to effectively aggregate information from relevant neighboring patches.
- Deformable convolutions complement patch attention: While patch attention captures global context, deformable convolutions handle local geometric distortions and fine details, creating a robust pipeline.
Key Results
- Quantitative Performance: PatchNet achieves state-of-the-art results on both BS-RSCD and Fastec-RS datasets. On BS-RSCD, it shows a 2.32dB improvement in average PSNR compared to the best combination of existing RS correction (JCD) and SR (EDSR) methods. It also outperforms methods that only perform RS correction on high-resolution inputs.
- Qualitative Performance: Visual results demonstrate that PatchNet produces significantly sharper and more detailed frames with fewer artifacts compared to baseline methods. The paper highlights that PatchNet's joint learning approach leads to visually superior results, especially in text regions.
- Patch Size Impact: Ablation studies show that an 8x8 patch size for attention yields the best performance, indicating a good balance between capturing local details and global context.
Limitations and Future Work
- Camera Configuration Dependency: The current model is trained on specific camera configurations and may not generalize perfectly to unseen settings.
- Future Directions: The authors suggest developing models that are agnostic to camera configurations by incorporating specific parameters like readout and exposure times. This would likely require semi-supervised or self-supervised learning approaches due to the difficulty of obtaining such data.
Summary updated