Bringing Rolling Shutter Images Alive with Dual Reversed Distortion

Zhihang Zhong, Mingdeng Cao, Xiao Sun, Zhirong Wu, Zhongyi Zhou, Yinqiang Zheng, Stephen Lin, Imari Sato

Abstract

Rolling shutter (RS) distortion occurs when a camera captures a scene row-by-row rather than all at once, leading to the well-known "jello effect" in dynamic scenes. While this distortion is typically viewed as a nuisance, it actually encodes temporal information about the scene's motion. We present IFED (Intermediate Frames Extractor using Dual RS images), a novel deep learning framework that leverages a dual-camera setup with reversed scanning directions to reconstruct high-quality, undistorted global shutter (GS) video sequences from distorted stills. By decoupling camera readout timing from scene velocity, our model achieves superior generalization across different hardware and complex motion patterns.

Introduction

Most modern smartphones and consumer-grade cameras utilize Rolling Shutter sensors due to their cost-effectiveness and high resolution. However, because each row of pixels is exposed at a slightly different time, any relative motion between the camera and the scene results in geometric warping. Straight lines appear slanted, and rotating objects appear unnaturally stretched or compressed.

Correcting RS distortion is a fundamentally ill-posed problem. Traditional methods attempting to use consecutive frames from a single camera often struggle to distinguish between camera motion and independent object motion. Furthermore, these models frequently overfit to the specific "readout speed" of the training hardware, failing when applied to cameras with different sensor characteristics. Our research addresses these challenges through a hardware-software co-design that utilizes dual reversed distortion.

The Dual Reversed Scanning Approach

Instead of relying on a single temporal sequence, we propose using a pair of images captured simultaneously by two RS cameras with opposite scanning directions: one scanning top-to-bottom (t2b) and the other scanning bottom-to-top (b2t).

This symmetry provides a powerful geometric constraint. Mathematically, determining camera motion from two standard consecutive RS frames requires at least 17 point matches. By contrast, the symmetric nature of dual reversed distortion reduces this requirement to just 5 point matches. This makes the problem significantly more constrained and solvable, even in the presence of complex dynamic elements.

The IFED Model Architecture

The core of our solution is the Intermediate Frames Extractor using Dual RS images (IFED). The model is designed to "unroll" the two distorted images into a sequence of undistorted GS frames. It consists of several key innovations:

1. The Time Cube Prior

Since the scanning mechanism of an RS camera is deterministic, we can pre-define a "Time Cube." This represents the known relative time gap between each row and the target global shutter timestamp. By treating time as a known prior, we remove a significant variable that usually confuses motion estimation models.

2. Velocity Cube Learning

Rather than predicting optical flow (direct pixel displacement) which is tied to specific readout speeds, IFED learns a Velocity Cube. The model estimates the velocity field of the scene; this velocity is then multiplied by the values in the Time Cube to derive the final optical flow. This decoupling is the secret to the model's hardware-agnostic nature, allowing it to function across various sensors without retraining.

3. Iterative Refinement and Merging

The system uses a sub-network called VelocityNet to refine the velocity field in a coarse-to-fine manner. Finally, an encoder-decoder network generates a mask and residual cube to merge the warped information from both cameras, effectively handling occlusions and preserving fine textures.

Experimental Results

To train and evaluate the model, we developed the RS-GOPRO dataset, synthesizing realistic dual reversed RS images from high-frame-rate (240 fps) video. We also validated the approach using a custom real-world rig equipped with a beam-splitter and two synchronized, inverted RS cameras.

  • Accuracy: IFED achieved a PSNR of 32.07 on the RS-GOPRO benchmark, significantly outperforming previous state-of-the-art methods like RSSR.
  • Generalization: The model demonstrated remarkable robustness. Despite being trained on a fixed readout setting (87µs per row), it successfully corrected images from sensors with readout speeds ranging from 65µs to 195µs with no loss in performance.
  • Real-World Utility: In practical tests involving high-speed motion, such as spinning fans or moving vehicles, IFED successfully recovered the true geometry of objects. Furthermore, we demonstrated that the undistorted sequences produced by IFED lead to significantly more accurate 3D reconstructions in Structure from Motion (SfM) pipelines compared to using raw RS images.

Conclusion and Future Work

IFED represents a significant step forward in computational photography, transforming the "flaw" of rolling shutter distortion into a source of temporal data. While the current requirement for a dual-camera rig with a beam-splitter is a hurdle for standard consumer use, the underlying principles of velocity decoupling and symmetry offer a roadmap for future mobile implementations. Future research will focus on adapting these techniques for the slightly offset stereo camera configurations found on modern smartphones and improving performance under extreme angular velocities.

Summary updated

Abstract

Rolling shutter (RS) distortion can be interpreted as the result of picking a row of pixels from instant global shutter (GS) frames over time during the exposure of the RS camera. This means that the information of each instant GS frame is partially, yet sequentially, embedded into the row-dependent distortion. Inspired by this fact, we address the challenging task of reversing this process, i.e., extracting undistorted GS frames from images suffering from RS distortion. However, since RS distortion is coupled with other factors such as readout settings and the relative velocity of scene elements to the camera, models that only exploit the geometric correlation between temporally adjacent images suffer from poor generality in processing data with different readout settings and dynamic scenes with both camera motion and object motion. In this paper, instead of two consecutive frames, we propose to exploit a pair of images captured by dual RS cameras with reversed RS directions for this highly challenging task. Grounded on the symmetric and complementary nature of dual reversed distortion, we develop a novel end-to-end model, IFED, to generate dual optical flow sequence through iterative learning of the velocity field during the RS time. Extensive experimental results demonstrate that IFED is superior to naive cascade schemes, as well as the state-of-the-art which utilizes adjacent RS images. Most importantly, although it is trained on a synthetic dataset, IFED is shown to be effective at retrieving GS frame sequences from real-world RS distorted images of dynamic scenes. Code is available at https://github.com/zzh-tech/Dual-Reversed-RS.
BibTeX
@inproceedings{zhong2022bringing,
  author = {Zhihang Zhong and Mingdeng Cao and Xiao Sun and Zhirong Wu and Zhongyi Zhou and Yinqiang Zheng and Stephen Lin and Imari Sato},
  title = {{Bringing Rolling Shutter Images Alive with Dual Reversed Distortion}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=TKhOFC7cwod},
}
Figure