Introduction
Most modern consumer cameras, particularly those in smartphones, utilize CMOS sensors with a rolling shutter (RS) mechanism. Unlike global shutter (GS) sensors that capture an entire scene simultaneously, RS sensors record the image row-by-row. When the camera or objects in the scene move during this sequential capture, the resulting images suffer from geometric distortions such as skew, wobble, or the "jello effect."
This paper introduces a novel deep learning framework designed for Rolling Shutter Temporal Super-Resolution (RSSR). The goal is to "invert" the rolling shutter effect by transforming two consecutive distorted RS images into a high-framerate, distortion-free GS video sequence.
Methodology: Geometry-Aware RSSR
Previous attempts to undistort RS images often treated the problem as a single-frame restoration task, which is mathematically ill-posed. The authors propose a geometry-aware pipeline built on three theoretical pillars that link RS distortion to motion dynamics.
1. Bidirectional RS Undistortion Flows
The framework formulates how pixels transition from their distorted positions in an RS frame to their "correct" positions in a virtual GS frame. This assumes a constant linear and angular velocity during the short exposure window of two frames.
2. The Scaling Connection
A key contribution of this work is the mathematical proof that RS undistortion flow is essentially a scaled version of standard optical flow. The scaling factor ($c$) is determined by the specific scanline (row) being corrected. This allows the model to leverage mature optical flow estimation techniques.
3. Scanline Interconversion
By developing a method to convert the undistortion flow of one scanline to another, the model can generate a GS frame for any arbitrary point in time represented by a scanline. Since a typical image has hundreds of rows, this enables the synthesis of extremely high-framerate video from just two input frames.
Network Architecture
The RSSR network employs a multi-stage approach:
- Optical Flow Estimation: A pre-trained PWC-Net identifies pixel-wise motion between the two input RS frames.
- Middle-scanline Estimator: A UNet-style backbone learns the necessary scaling factors and refines the flow fields.
- Softmax Splatting: A differentiable warping technique synthesizes the final GS images based on the calculated flows, ensuring high visual fidelity.
Experimental Results
The model was rigorously evaluated on two benchmarks: Carla-RS (synthetic autonomous driving data) and Fastec-RS (real-world scenes synthesized from high-speed GS footage).
- Superior Quality: On Carla-RS, the RSSR method achieved a PSNR of 30.17, significantly outperforming the previous state-of-the-art, DeepUnrollNet (26.90). Qualitatively, the results show sharper edges and fewer ghosting artifacts.
- Computational Efficiency: The inference process is nearly real-time, taking approximately 0.12 seconds to restore a single GS frame. A massive 960-frame GS video sequence can be generated in just 1.8 seconds, whereas traditional geometric methods can take hundreds of seconds per frame.
Conclusion and Future Work
By baking geometric constraints into a deep learning architecture, the authors have created a robust and efficient solution for RS distortion. While the method currently relies on a constant velocity assumption and can struggle with complex occlusions, it represents a significant step forward in temporal super-resolution. Future work may explore non-constant motion models and more sophisticated occlusion handling to improve performance in highly dynamic environments.
Summary updated