Introduction
Fisheye lenses are widely utilized in surveillance, robotics, and automotive applications due to their expansive field of view. However, this benefit comes at the cost of extreme radial "barrel" distortion, which significantly degrades the performance of standard computer vision algorithms designed for perspective cameras. While single-image rectification has seen significant progress, applying these techniques to video sequences frame-by-frame often results in severe temporal instability, manifesting as nauseating jitter or "wobbling" artifacts. This paper presents a novel multi-stage deep learning framework that leverages spatiotemporal deformation perception to achieve smooth, stable, and accurate fisheye video rectification.
Problem Statement
The primary challenge in fisheye video rectification is maintaining temporal consistency while accurately correcting local and global distortions. Independent frame processing fails because small variations in the model's predictions across consecutive frames lead to temporal discontinuities. Furthermore, existing methods often struggle to preserve fine local details or maintain the global geometric structure of the scene, especially near the edges of the lens where distortion is most severe.
Proposed Methodology
The authors introduce a comprehensive architecture designed to perceive and aggregate deformation information across both spatial and temporal dimensions. The pipeline consists of four primary components:
- Flow Estimation Network (FSN): This stage processes a window of five consecutive frames to predict an initial "intra-frame optical flow," which serves as a mapping for pixel-wise rectification.
- Temporal Weighting Scheme (TWS): To mitigate jitter, a progressive weighting system is employed. By prioritizing previous flow predictions, the network ensures that rectification parameters evolve smoothly over time.
- Temporal Deformation Aggregator (TDA): Utilizing 3D convolutions, this module identifies correlations between frames. It exploits the fact that objects may appear less distorted at certain points in their trajectory (e.g., the center of the lens), using this information to improve rectification when the object moves to more distorted regions.
- Spatial Deformation Perception Module (SDM): This module represents the core technical innovation. It utilizes the mathematical relationship between motion in distorted space and motion in rectified space. By comparing these "dual flows," the network can precisely identify and correct local spatial inaccuracies.
The final stage is a Flow Enhancement Network that integrates features from the TDA and SDM to produce a refined global optical flow, which is then used to resample the original fisheye frames into a stabilized video output.
Key Insights and Ideas
- Motion as a Geometric Signal: A key breakthrough of this work is the realization that inter-frame motion (optical flow) provides a direct signal for lens distortion. Deviations from expected linear motion in a rectified space reveal the underlying curvature of the lens distortion.
- Temporal Context for Structural Consistency: By analyzing a five-frame window, the model treats the video as a continuous event rather than a series of isolated images, allowing it to maintain better structural integrity.
- Progressive Smoothing: The TWS effectively acts as a temporal low-pass filter, denoising the rectification parameters to eliminate the high-frequency jitter common in per-frame methods.
Experimental Results
The proposed method was evaluated against state-of-the-art (SOTA) models, including DDM and PCN, using synthesized datasets derived from DAVIS and Youtube-VOS, as well as real-world fisheye footage.
Quantitative Performance
On the Youtube-VOS dataset, the model achieved a PSNR of 28.18 and an SSIM of 0.8889, significantly outperforming the previous best (DDM), which achieved 26.42 and 0.7948, respectively. Furthermore, the method attained a stability score of 0.895, confirming its effectiveness in reducing temporal jitter.
Qualitative Analysis
Visual comparisons demonstrate that the proposed method produces sharper textures and more accurate geometric reconstructions. Unlike GAN-based approaches that often introduce wavy artifacts, this model maintains straight lines even at the extreme edges of the frame.
Conclusion and Future Work
This research successfully demonstrates that spatiotemporal perception is critical for high-quality fisheye video rectification. By leveraging the relationship between motion and distortion, the proposed framework achieves superior accuracy and stability. Future work will focus on optimizing the computational efficiency of the 3D convolutions and attention mechanisms to enable real-time processing for applications such as live drone navigation and augmented reality.
Summary updated