Abstract
Rolling shutter (RS) distortion occurs when a camera captures a scene row-by-row rather than all at once, leading to the well-known "jello effect" in dynamic scenes. While this distortion is typically viewed as a nuisance, it actually encodes temporal information about the scene's motion. We present IFED (Intermediate Frames Extractor using Dual RS images), a novel deep learning framework that leverages a dual-camera setup with reversed scanning directions to reconstruct high-quality, undistorted global shutter (GS) video sequences from distorted stills. By decoupling camera readout timing from scene velocity, our model achieves superior generalization across different hardware and complex motion patterns.
Introduction
Most modern smartphones and consumer-grade cameras utilize Rolling Shutter sensors due to their cost-effectiveness and high resolution. However, because each row of pixels is exposed at a slightly different time, any relative motion between the camera and the scene results in geometric warping. Straight lines appear slanted, and rotating objects appear unnaturally stretched or compressed.
Correcting RS distortion is a fundamentally ill-posed problem. Traditional methods attempting to use consecutive frames from a single camera often struggle to distinguish between camera motion and independent object motion. Furthermore, these models frequently overfit to the specific "readout speed" of the training hardware, failing when applied to cameras with different sensor characteristics. Our research addresses these challenges through a hardware-software co-design that utilizes dual reversed distortion.
The Dual Reversed Scanning Approach
Instead of relying on a single temporal sequence, we propose using a pair of images captured simultaneously by two RS cameras with opposite scanning directions: one scanning top-to-bottom (t2b) and the other scanning bottom-to-top (b2t).
This symmetry provides a powerful geometric constraint. Mathematically, determining camera motion from two standard consecutive RS frames requires at least 17 point matches. By contrast, the symmetric nature of dual reversed distortion reduces this requirement to just 5 point matches. This makes the problem significantly more constrained and solvable, even in the presence of complex dynamic elements.
The IFED Model Architecture
The core of our solution is the Intermediate Frames Extractor using Dual RS images (IFED). The model is designed to "unroll" the two distorted images into a sequence of undistorted GS frames. It consists of several key innovations:
1. The Time Cube Prior
Since the scanning mechanism of an RS camera is deterministic, we can pre-define a "Time Cube." This represents the known relative time gap between each row and the target global shutter timestamp. By treating time as a known prior, we remove a significant variable that usually confuses motion estimation models.
2. Velocity Cube Learning
Rather than predicting optical flow (direct pixel displacement) which is tied to specific readout speeds, IFED learns a Velocity Cube. The model estimates the velocity field of the scene; this velocity is then multiplied by the values in the Time Cube to derive the final optical flow. This decoupling is the secret to the model's hardware-agnostic nature, allowing it to function across various sensors without retraining.
3. Iterative Refinement and Merging
The system uses a sub-network called VelocityNet to refine the velocity field in a coarse-to-fine manner. Finally, an encoder-decoder network generates a mask and residual cube to merge the warped information from both cameras, effectively handling occlusions and preserving fine textures.
Experimental Results
To train and evaluate the model, we developed the RS-GOPRO dataset, synthesizing realistic dual reversed RS images from high-frame-rate (240 fps) video. We also validated the approach using a custom real-world rig equipped with a beam-splitter and two synchronized, inverted RS cameras.
- Accuracy: IFED achieved a PSNR of 32.07 on the RS-GOPRO benchmark, significantly outperforming previous state-of-the-art methods like RSSR.
- Generalization: The model demonstrated remarkable robustness. Despite being trained on a fixed readout setting (87µs per row), it successfully corrected images from sensors with readout speeds ranging from 65µs to 195µs with no loss in performance.
- Real-World Utility: In practical tests involving high-speed motion, such as spinning fans or moving vehicles, IFED successfully recovered the true geometry of objects. Furthermore, we demonstrated that the undistorted sequences produced by IFED lead to significantly more accurate 3D reconstructions in Structure from Motion (SfM) pipelines compared to using raw RS images.
Conclusion and Future Work
IFED represents a significant step forward in computational photography, transforming the "flaw" of rolling shutter distortion into a source of temporal data. While the current requirement for a dual-camera rig with a beam-splitter is a hurdle for standard consumer use, the underlying principles of velocity decoupling and symmetry offer a roadmap for future mobile implementations. Future research will focus on adapting these techniques for the slightly offset stereo camera configurations found on modern smartphones and improving performance under extreme angular velocities.
Summary updated