TL;DR
This paper introduces SensorFlow, a novel video stabilization framework that fuses sensor data (IMU, OIS) with image-based optical flow to achieve robust, high-quality stabilization. It tackles the limitations of existing methods by introducing sensor-based pre-stabilization, an occlusion-aware 3D CNN for dense warp field estimation, and a novel masking strategy to handle dynamic objects and disocclusions.
What Problem Does This Paper Address?
Video stabilization is crucial for producing professional-looking footage, but it's a challenging task. Traditional methods often rely on tracking features or optical flow, which can struggle with complex camera motions, dynamic scenes, and disocclusions (where parts of the scene become visible or hidden). While sensor-based methods (like those using gyroscopes) are robust to large motions, they are often limited to rotational stabilization and susceptible to calibration errors. Existing sensor-image fusion methods often use simplified motion models that fail to capture residual motion and parallax accurately. This paper aims to bridge this gap by creating a robust and high-quality stabilization system that leverages the strengths of both sensor and image-based approaches, particularly in challenging dynamic scenarios.
Methods
SensorFlow employs a two-stage pipeline:
- Sensor-based Pre-stabilization: This stage uses gyroscope and OIS sensor data to smooth out large-scale camera rotations and minor translations. A key innovation here is an angular velocity domain optimization that makes the path smoothing frame-rate invariant.
- Image-based Refinement: The pre-stabilized frames are then fed into a novel 3D convolutional neural network (CNN). This CNN infers dense warp fields to eliminate residual translational motion and jitter. To handle challenging aspects like dynamic objects and disocclusions, the paper introduces:
- An occlusion-aware loss during training.
- A novel masking scheme that combines flow cycle consistency and fundamental matrix outlier detection to identify invalid optical flow regions. These regions are then inpainted using spatially smooth flow vectors.
Datasets
- Training: The RealEstate10K dataset was used for training. While it primarily contains smooth camera motions and static scenes, the authors augmented it with random affine transformations to simulate complex optical flows and introduce occlusions.
- Evaluation: The Shi et al. [31] test set, which includes sensor data, was used for evaluation. The NUS dataset was also used for evaluating general video stabilization without sensor information.
Key Insights and Ideas
- Angular Velocity for Smoothing: Using angular velocity for smoothing camera paths, rather than directly smoothing quaternions, offers a more robust and frame-rate invariant approach.
- 3D CNN for Temporal and Spatial Consistency: A 3D CNN is crucial for video stabilization because it can simultaneously consider spatial relationships within frames and temporal dependencies across frames, leading to smoother and more coherent warp fields.
- Masking is Crucial for Robustness: Simply relying on optical flow can lead to artifacts, especially with dynamic objects and disocclusions. The proposed masking strategy effectively identifies and handles these problematic regions, significantly improving visual quality.
- Occlusion-Aware Training: Explicitly penalizing unreliable optical flow regions during training makes the network more robust to challenging scenarios without relying solely on the masking step.
- Improved Stability Metric: The authors propose a new stability metric that focuses on inter-frame motion and the DC component of the FFT, which better reflects perceived stability by removing the influence of intentional motion.
Key Results
- Qualitative: SensorFlow produces visually pleasing results, outperforming optical flow-based methods that often introduce distortions around dynamic objects. It also surpasses non-traditional methods like interpolation and 3D stabilization in robustness.
- Quantitative:
- On the Shi et al. [31] dataset, SensorFlow achieved significantly better stability (+15.0% compared to the second best) with comparable distortion and cropping.
- When tested on general videos (NUS dataset) without sensor data, it still achieved the most stable results with minimal FOV cropping.
- A user study showed that participants preferred SensorFlow over existing methods in over 90% of cases.
Limitations and Future Work
- Reliance on Optical Flow: The pipeline's dependence on optical flow means it's susceptible to artifacts from potentially inaccurate flows.
- Limited Parallax Correction: The current 2D warp field approach has limitations in correcting parallax. The authors suggest that novel view synthesis with 3D dynamic scene representation would be ideal.
- Real-time Performance: While the current implementation is not real-time, the authors believe it can be adapted for online or real-time stabilization with further optimizations, leaving this as a direction for future research.
Summary updated