TL;DR
This paper introduces a novel, real-time video stabilization method for mobile devices that leverages gyroscope data to correct for camera shake and rolling shutter distortion. It achieves smoother video by using a non-linear filter that can flatten small motions entirely while still smoothly following intentional camera movements, outperforming previous gyroscope-based methods.
What Problem Does This Paper Address?
Modern smartphones are ubiquitous for casual video recording, but the resulting footage is often plagued by camera shake due to handheld operation. This shake can be distracting or even make videos unwatchable. Furthermore, many mobile cameras use rolling shutter sensors, which introduce distortions like shearing or stretching when the camera moves during exposure. While image-based stabilization methods exist, they can be computationally expensive and struggle with scenes lacking distinct features or with significant foreground motion. This paper tackles the challenge of providing effective, real-time video stabilization and rolling shutter correction directly on mobile devices, minimizing processing overhead and battery drain.
Methods
The core of the proposed method is a non-linear filter that processes gyroscope data to estimate camera orientation. Unlike traditional linear filters that smooth all motion, this non-linear approach aims to keep the virtual camera static when small, unintended motions are detected, effectively eliminating jitter. For intentional camera movements, it smoothly follows the desired motion.
Key aspects of the method include:
- Gyroscope-based Motion Tracking: The system relies on the device's 3-axis gyroscope to track angular velocity, which is significantly more computationally efficient than image-based tracking.
- Non-Linear Filtering: The filter is designed to distinguish between intentional and unintentional motion. It can completely flatten small, high-frequency motions (camera shake) while allowing for smooth, low-frequency intentional movements. This is achieved by using a mixing weight that adapts based on how much the "crop polygon" (the stable region of the frame) is within a defined "inner region" of the image.
- Rolling Shutter Correction: The method integrates rolling shutter correction by dividing the crop polygon into slices and applying different projective warps to each slice, based on the gyroscope's precise temporal orientation data.
- Look-ahead Buffer: An optional look-ahead buffer of a few frames can be used to predict future motion, allowing for smoother transitions and preemptive adjustments to the camera path.
- Real-time Implementation: The algorithm is designed to run within the camera capture pipeline, processing each frame as it arrives with minimal buffering, making it suitable for live preview and recording.
Datasets
Not discussed in the paper. The authors mention recording a series of videos on a prototype Android tablet and an iPhone 5 for evaluation.
Key Insights and Ideas
- Non-linearity is Key for Jitter Removal: The crucial insight is that a non-linear filter can achieve a more complete elimination of camera shake by treating small motions differently from larger, intentional ones. This allows for truly static segments in the video.
- Gyroscope Efficiency: Leveraging the gyroscope for motion estimation drastically reduces computational cost, memory bandwidth, and battery consumption compared to image-based methods, making it ideal for mobile devices.
- Rolling Shutter Correction with Gyroscope: The precise temporal information from the gyroscope is essential for accurately correcting rolling shutter distortions on a per-frame and even per-scanline basis.
- Adaptive Filtering for Smoothness: The adaptive mixing of input velocity with the virtual camera's velocity, controlled by the crop polygon's position, allows the system to maintain stability while gracefully following intentional movements.
Key Results
- The proposed method effectively eliminates high-frequency jitter, making videos appear as if shot on a tripod.
- It provides a similar level of stabilization quality to state-of-the-art feature-based methods but at a fraction of the processing cost.
- The real-time implementation on an Android tablet runs the filter in 160 microseconds and the image warp in 15 ms, demonstrating its efficiency.
- Qualitative comparisons show results comparable to YouTube's stabilization tool.
- The method successfully corrects rolling shutter distortions, leading to sharper images.
Limitations and Future Work
- Translation Tracking: The current method relies on gyroscopes, which do not track translational motion. This can lead to limitations in scenes with significant camera translation, potentially causing distortions near occlusions.
- Fixed Crop Window: The algorithm uses a fixed-size crop window. The authors suggest that varying the crop size dynamically could improve stabilization.
- Future Directions:
- Investigate using accelerometers to track small translations.
- Explore dynamic crop window sizing.
- Incorporate visual tracking of sparse features, leveraging the already accounted-for rotation.
- Store gyroscope readings as a separate track in the output video for potential offline stabilization.
Summary updated