Most geometric algorithms in computer vision operate under a convenient fiction: the global shutter assumption. This assumption posits that an entire image frame records a single, instantaneous snapshot of the world. However, nearly all modern consumer cameras, smartphones, and robotics platforms use CMOS sensors with a rolling shutter.
Instead of exposing the entire sensor at once, a rolling shutter sensor captures the image sequentially, row by row. This introduces a tiny but deterministic temporal delay between the top and bottom of the frame. When the camera or the scene moves rapidly during this readout window, this sequential exposure maps different points in time to different rows of the same image. The result is a suite of complex geometric distortions—such as skewing, tilting, and the characteristic "jello effect"—that break standard spatial assumptions and degrade downstream tasks like 3D reconstruction, visual odometry, and stabilization.
The Geometry of Time-Slit Imaging
To understand why rolling shutter (RS) distortion occurs, it is helpful to view an image not as a static 2D grid, but as a curved slice through a 3D space-time volume. As explained by Eyal Naor et al. in Combining Internal and External Constraints for Unrolling Shutter in Videos, traditional 2D spatial corrections fail because they ignore this underlying temporal mechanism.
When a camera is stationary, its space-time volume is uniform, and a rolling shutter capture yields a perfectly flat temporal slice—identical to a global shutter (GS) image. But when the camera rotates or translates during exposure, the sensor effectively sweeps a 1D slit through a changing 3D space-time volume. As noted by Peidong Liu et al. in Deep Shutter Unrolling Network, the vertical spatial axis ($y$) becomes inextricably linked with the temporal axis ($t$). A straight vertical pillar in the real world becomes a diagonal line in the image because the bottom of the pillar is captured fractions of a second later than the top, after the camera has moved.
Global Shutter: [t0, t0, t0, t0, t0] <-- All rows share one timestamp
Rolling Shutter: [t0, t1, t2, t3, t4] <-- Vertical axis maps directly to time
This coupling means that a single RS frame does not possess a single camera pose. Instead, it contains a continuous trajectory of poses.
Continuous-Time Trajectories and Absolute Pose
Because standard geometric solvers assume a single camera pose per frame, they fail catastrophically when presented with RS data. To resolve this, researchers must explicitly model the sensor's kinematics during the readout period.
One approach is to calibrate the exact temporal characteristics of the sensor. Luc Oth et al. demonstrated in Rolling Shutter Camera Calibration that the "line delay"—the precise time gap between successive row exposures—can be recovered by modeling the camera's trajectory as a continuous-time B-spline. By parameterizing the motion continuously, every individual row can be mapped to its exact position along the camera's path.
Once the temporal parameters are known, geometric estimation requires new minimal solvers. In R6P - Rolling shutter absolute pose problem, Cenek Albl developed a non-iterative solution that computes the absolute pose and camera velocity using just six 2D-to-3D point correspondences. By integrating the camera's linear and angular velocities directly into the projection equations, the solver accounts for the row-wise displacement during the estimation process itself.
An ingenious hardware-software alternative involves changing how the data is captured. Cenek Albl et al. showed in From Two Rolling Shutters to One Global Shutter that mounting two rolling shutter cameras with different readout directions (e.g., one reading top-to-bottom and the other bottom-to-top or left-to-right) creates complementary geometric distortions. Because the same physical point is captured at two different relative timestamps by the two sensors, the geometric discrepancy between the two distorted images uniquely constrains the camera's motion, allowing the algorithm to mathematically synthesize a clean, virtual global shutter perspective.
Learning to Unroll the Shutter
While explicit geometric modeling works well for rigid scenes, real-world video often contains non-rigid object motion compounded by motion blur. To address these complex interactions, modern approaches leverage deep neural networks to invert the rolling shutter process.
For single-image correction, the network must infer a continuous motion field from a static, distorted input. Vijay Rengarajan et al. designed a specialized architecture in Unrolling the Shutter: CNN to Correct Motion Distortions that employs long rectangular convolutional kernels. These kernels are explicitly tailored to match the row-wise nature of the sensor readout, allowing the network to efficiently analyze and reverse the vertical-to-temporal transformations.
When multiple frames are available, temporal consistency provides a powerful constraint. The architecture proposed by Bin Fan et al. in SUNet: Symmetric Undistortion Network for Rolling Shutter Correction uses a symmetric network structure to process two consecutive distorted frames. By analyzing the motion flow forward and backward, the network interpolates a geometrically sound global shutter frame at the exact temporal midpoint between the two captures.
In real-world settings, long exposure times introduce motion blur alongside rolling shutter artifacts. As a result, each row exhibits a unique, spatially varying blur kernel. Yinqiang Zheng et al. addressed this joint degradation in Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes by utilizing a three-stream network with bi-directional warping and deformable attention, untangling the "jello effect" from motion blur by learning the physical coupling between exposure duration and row readout timing.
Temporal Alignment in Neural Rendering
The rise of neural scene representations, such as 3D Gaussian Splatting (3DGS), has brought the rolling shutter problem to the forefront of real-time rendering and autonomous driving. When sensors are mounted on fast-moving vehicles, ignoring the row-wise timestamp causes severe ghosting artifacts and incorrect spatial mapping.
To maintain geometric consistency, modern neural renderers integrate the rolling shutter model directly into the rasterization pipeline. In SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving, Georg Hess et al. introduced a mechanism that dynamically adjusts the spatial positions of 3D primitives during rendering. By shifting the mean position of each Gaussian primitive based on the exact timestamp of the specific pixel ray being processed, the model prevents volume carving errors.
Similarly, Qi Wu et al. developed 3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting, which leverages the Unscented Transform to project individual sample points within a Gaussian using time-dependent camera poses.
Standard Rendering: [3D Scene] -> [Single Camera Pose] -> [Distorted Frame]
RS-Aware Rendering: [3D Scene] -> [Pose(t) per Row/Ray] -> [Clean Frame]
By embedding the continuous-time camera trajectory directly into the forward projection pass, the rendering engine naturally unwarps the motion-induced artifacts, achieving high-fidelity reconstructions without requiring a separate post-processing rectification step.
Ultimately, whether through explicit geometric solvers, deep multi-frame warping, or time-dependent neural rendering, the core insight remains the same: correcting rolling shutter distortion requires treating the image sensor not as a spatial canvas captured at an instant, but as a continuous collector of space-time data.
Key Foundational Work
Geometric Models of Rolling-Shutter Cameras (2005) — contributed 8 concept elements to this direction.