Introduction
Rolling Shutter (RS) sensors, ubiquitous in modern mobile devices and drones, capture images row-by-row. This sequential acquisition introduces geometric distortions—often called the "jello effect"—when either the camera or the subject is in motion. Traditionally, computer vision pipelines mitigate these artifacts through computationally expensive Image Signal Processing (ISP) correction steps. This paper investigates whether such correction is strictly necessary for modern deep learning models or if these models can inherently learn to compensate for RS distortion.
Methodology: Synthetic Paired Datasets
A primary challenge in studying RS effects is the lack of real-world datasets with perfectly aligned Global Shutter (GS) and Rolling Shutter (RS) image pairs. To overcome this, the researchers utilized Unreal Engine 5 (UE5) to build a high-fidelity "Sample City" with 40 distinct urban scenes.
The team generated two primary benchmarks:
- Normal Speed: Pedestrians moving at 2 m/s.
- High Speed: Pedestrians moving at 20 m/s to exaggerate RS artifacts.
By rendering scenes at extremely high frame rates and selectively stitching rows, they simulated the line-by-line scan of an RS sensor, creating a first-of-its-kind paired dataset for training and evaluating state-of-the-art models like YOLOv8 and DETR.
Key Findings: The Coarse vs. Fine Localization Gap
The study revealed a nuanced impact of shutter distortion on model performance:
- Coarse Detection Robustness: For standard detection tasks ([email protected]), the models showed surprising resilience. YOLOv8 achieved an mAP of 0.82 on both GS and RS images at normal walking speeds, suggesting that the "identity" of a pedestrian is preserved despite minor tilting.
- Fine-Grained Localization Failure: The impact became severe when high precision was required. On the [email protected]:0.95 metric, YOLOv8's performance dropped by 26% (from 0.60 to 0.44) on RS data. The distortion prevents the model from accurately defining the object's boundaries.
Implicit Compensation and Pipeline Efficiency
A critical insight from the research is that models can be "taught" to ignore or compensate for RS artifacts. By training on a mixture of GS and RS data, the test mAP on distorted images improved from 0.63 to 0.80. This suggests that the "correction" typically performed by the ISP can be offloaded to the neural network's weights.
This finding paves the way for "ISP-less" pipelines, which are significantly more energy-efficient for edge devices like AR glasses or battery-constrained drones. Instead of spending power to rectify an image, the model processes the raw, distorted input directly.
Conclusion
While rolling shutter distortion significantly degrades fine-grained localization, its impact on general object detection is manageable. By leveraging synthetic datasets for training, developers can create robust models that eliminate the need for power-hungry ISP correction steps, moving toward more efficient, end-to-end autonomous systems.
Summary updated