Inverting a Rolling Shutter Camera: Bring Rolling Shutter Images to High Framerate Global Shutter Video

Bin Fan, Yuchao Dai

Introduction

Most modern consumer cameras, particularly those in smartphones, utilize CMOS sensors with a rolling shutter (RS) mechanism. Unlike global shutter (GS) sensors that capture an entire scene simultaneously, RS sensors record the image row-by-row. When the camera or objects in the scene move during this sequential capture, the resulting images suffer from geometric distortions such as skew, wobble, or the "jello effect."

This paper introduces a novel deep learning framework designed for Rolling Shutter Temporal Super-Resolution (RSSR). The goal is to "invert" the rolling shutter effect by transforming two consecutive distorted RS images into a high-framerate, distortion-free GS video sequence.

Overview of the Rolling Shutter imaging mechanism and the RSSR pipeline which converts distorted RS images into a clean GS video sequence.

Methodology: Geometry-Aware RSSR

Previous attempts to undistort RS images often treated the problem as a single-frame restoration task, which is mathematically ill-posed. The authors propose a geometry-aware pipeline built on three theoretical pillars that link RS distortion to motion dynamics.

1. Bidirectional RS Undistortion Flows

The framework formulates how pixels transition from their distorted positions in an RS frame to their "correct" positions in a virtual GS frame. This assumes a constant linear and angular velocity during the short exposure window of two frames.

2. The Scaling Connection

A key contribution of this work is the mathematical proof that RS undistortion flow is essentially a scaled version of standard optical flow. The scaling factor ($c$) is determined by the specific scanline (row) being corrected. This allows the model to leverage mature optical flow estimation techniques.

3. Scanline Interconversion

By developing a method to convert the undistortion flow of one scanline to another, the model can generate a GS frame for any arbitrary point in time represented by a scanline. Since a typical image has hundreds of rows, this enables the synthesis of extremely high-framerate video from just two input frames.

Visual comparison showing how standard optical flow relates to RS undistortion flows at the first, middle, and last scanlines, illustrating the scaling connection.

Network Architecture

The RSSR network employs a multi-stage approach:

  • Optical Flow Estimation: A pre-trained PWC-Net identifies pixel-wise motion between the two input RS frames.
  • Middle-scanline Estimator: A UNet-style backbone learns the necessary scaling factors and refines the flow fields.
  • Softmax Splatting: A differentiable warping technique synthesizes the final GS images based on the calculated flows, ensuring high visual fidelity.

Experimental Results

The model was rigorously evaluated on two benchmarks: Carla-RS (synthetic autonomous driving data) and Fastec-RS (real-world scenes synthesized from high-speed GS footage).

  • Superior Quality: On Carla-RS, the RSSR method achieved a PSNR of 30.17, significantly outperforming the previous state-of-the-art, DeepUnrollNet (26.90). Qualitatively, the results show sharper edges and fewer ghosting artifacts.
  • Computational Efficiency: The inference process is nearly real-time, taking approximately 0.12 seconds to restore a single GS frame. A massive 960-frame GS video sequence can be generated in just 1.8 seconds, whereas traditional geometric methods can take hundreds of seconds per frame.

Conclusion and Future Work

By baking geometric constraints into a deep learning architecture, the authors have created a robust and efficient solution for RS distortion. While the method currently relies on a constant velocity assumption and can struggle with complex occlusions, it represents a significant step forward in temporal super-resolution. Future work may explore non-constant motion models and more sophisticated occlusion handling to improve performance in highly dynamic environments.

Summary updated

Abstract

Rolling shutter (RS) images can be viewed as the result of the row-wise combination of global shutter (GS) images captured by a virtual moving GS camera over the period of camera readout time. The RS effect brings tremendous difficulties for the downstream applications. In this paper, we propose to invert the above RS imaging mechanism, i.e., recovering a high framerate GS video from consecutive RS images to achieve RS temporal super-resolution (RSSR). This extremely challenging problem, e.g., recovering 1440 GS images from two 720-height RS images, is far from being solved end-to-end. To address this challenge, we exploit the geometric constraint in the RS camera model, thus achieving geometry-aware inversion. Specifically, we make three contributions in resolving the above difficulties: (i) formulating the bidirectional RS undistortion flows under the constant velocity motion model, (ii) building the connection between the RS undistortion flow and optical flow via a scaling operation, and (iii) developing a mutual conversion scheme between varying RS undistortion flows that correspond to different scanlines. Building upon these formulations, we propose the first RS temporal super-resolution network in a cascaded structure to extract high framerate global shutter video. Our method explores the underlying spatio-temporal geometric relationships within a deep learning framework, where no extra supervision besides the middle-scanline ground truth GS image is needed. Essentially, our method can be very efficient for explicit propagation to generate GS images under any scanline. Experimental results on both synthetic and real data show that our method can produce high-quality GS image sequences with rich details, outperforming state-of-the-art methods.
BibTeX
@inproceedings{fan2021inverting,
  author = {Bin Fan and Yuchao Dai},
  title = {{Inverting a Rolling Shutter Camera: Bring Rolling Shutter Images to High Framerate Global Shutter Video}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00419},
}
Figure