Abstract
Rolling shutter (RS) distortion, commonly referred to as the "jello effect," is a prevalent artifact in CMOS sensor images caused by sequential row-by-row exposure during camera motion. While multi-frame methods leverage temporal information for correction, single-image RS rectification remains a challenging ill-posed problem. This paper introduces the Deep Homography Mixture (HM), a novel framework that models camera motion as a mixture of learned motion bases. By employing Gaussian-weighted image blocks and motion priors extracted from natural video sequences, the proposed method achieves state-of-the-art performance, significantly improving geometric integrity in diverse and challenging environments.
Introduction
Most modern smartphone cameras utilize CMOS sensors that capture images row-by-row. When the camera moves during this exposure period, each row records the scene from a slightly different perspective, leading to distortions such as skewed objects and bent lines. Correcting these artifacts from a single image is difficult because the underlying camera motion is unknown and the solution space is virtually infinite. Traditional single-frame methods often rely on geometric priors, such as the presence of straight lines, which limits their effectiveness in texture-poor or natural scenes like forests or night skies.
Methodology
The core innovation of this work is the Deep Homography Mixture (HM) model. Rather than attempting to predict per-pixel or per-row motion directly, the authors simplify the problem through three key mechanisms:
- Block-based Homography: The image is divided into $k$ horizontal blocks. Each block is assigned a homography transformation that describes the local camera rotation and translation.
- Gaussian Weighting: To ensure a physically plausible, continuous motion field and prevent visual artifacts at block boundaries, Gaussian weights are used to smoothly blend transformations across scanlines.
- Motion Basis Learning: The network predicts coefficients for a set of pre-learned motion bases instead of raw motion parameters. These bases are extracted via Singular Value Decomposition (SVD) from thousands of natural video sequences, effectively constraining the model to realistic camera trajectories.
The architecture utilizes a VGG-style backbone to process a single RS image and output the mixing weights for these bases across the image blocks.
Experimental Results
The model was rigorously evaluated on the Building Dataset, Carla-RS, and a new RS-Homo dataset designed to test robustness in adverse conditions such as low light and complex foregrounds.
Quantitative Performance
On the Building Dataset, the Deep Homography Mixture model achieved a PSNR of 33.34 dB and an SSIM of 0.75, outperforming previous state-of-the-art learning-based methods. Notably, the approach reduced the motion Root Mean Square Error (RMSE) by 56% compared to existing single-frame solutions. The model is also highly efficient, with an inference time of approximately 21.9ms, making it suitable for mobile applications.
Qualitative Analysis
Visually, the HM model produces significantly straighter contours and fewer artifacts in occluded areas compared to traditional methods. It excels in scenes where line-based priors are unavailable, demonstrating the power of its learned global motion structure.
Conclusion and Future Work
The Deep Homography Mixture framework provides a robust and efficient solution for single-image rolling shutter correction by leveraging learned motion priors. While the current model assumes a relatively planar scene, it sets a new benchmark for geometric restoration. Future research may explore incorporating depth estimation to handle scenes with extreme parallax or developing more complex motion models beyond homographies to further enhance 3D scene rectification.
Summary updated