Spatiotemporal Deformation Perception for Fisheye Video Rectification

Shangrong Yang, Chunyu Lin, Kang Liao, Yao Zhao

Introduction

Fisheye lenses are widely utilized in surveillance, robotics, and automotive applications due to their expansive field of view. However, this benefit comes at the cost of extreme radial "barrel" distortion, which significantly degrades the performance of standard computer vision algorithms designed for perspective cameras. While single-image rectification has seen significant progress, applying these techniques to video sequences frame-by-frame often results in severe temporal instability, manifesting as nauseating jitter or "wobbling" artifacts. This paper presents a novel multi-stage deep learning framework that leverages spatiotemporal deformation perception to achieve smooth, stable, and accurate fisheye video rectification.

Problem Statement

The primary challenge in fisheye video rectification is maintaining temporal consistency while accurately correcting local and global distortions. Independent frame processing fails because small variations in the model's predictions across consecutive frames lead to temporal discontinuities. Furthermore, existing methods often struggle to preserve fine local details or maintain the global geometric structure of the scene, especially near the edges of the lens where distortion is most severe.

Proposed Methodology

The authors introduce a comprehensive architecture designed to perceive and aggregate deformation information across both spatial and temporal dimensions. The pipeline consists of four primary components:

  1. Flow Estimation Network (FSN): This stage processes a window of five consecutive frames to predict an initial "intra-frame optical flow," which serves as a mapping for pixel-wise rectification.
  2. Temporal Weighting Scheme (TWS): To mitigate jitter, a progressive weighting system is employed. By prioritizing previous flow predictions, the network ensures that rectification parameters evolve smoothly over time.
  3. Temporal Deformation Aggregator (TDA): Utilizing 3D convolutions, this module identifies correlations between frames. It exploits the fact that objects may appear less distorted at certain points in their trajectory (e.g., the center of the lens), using this information to improve rectification when the object moves to more distorted regions.
  4. Spatial Deformation Perception Module (SDM): This module represents the core technical innovation. It utilizes the mathematical relationship between motion in distorted space and motion in rectified space. By comparing these "dual flows," the network can precisely identify and correct local spatial inaccuracies.
Overview of the proposed multi-stage network architecture for fisheye video rectification.

The final stage is a Flow Enhancement Network that integrates features from the TDA and SDM to produce a refined global optical flow, which is then used to resample the original fisheye frames into a stabilized video output.

Key Insights and Ideas

  • Motion as a Geometric Signal: A key breakthrough of this work is the realization that inter-frame motion (optical flow) provides a direct signal for lens distortion. Deviations from expected linear motion in a rectified space reveal the underlying curvature of the lens distortion.
  • Temporal Context for Structural Consistency: By analyzing a five-frame window, the model treats the video as a continuous event rather than a series of isolated images, allowing it to maintain better structural integrity.
  • Progressive Smoothing: The TWS effectively acts as a temporal low-pass filter, denoising the rectification parameters to eliminate the high-frequency jitter common in per-frame methods.

Experimental Results

The proposed method was evaluated against state-of-the-art (SOTA) models, including DDM and PCN, using synthesized datasets derived from DAVIS and Youtube-VOS, as well as real-world fisheye footage.

Quantitative Performance

On the Youtube-VOS dataset, the model achieved a PSNR of 28.18 and an SSIM of 0.8889, significantly outperforming the previous best (DDM), which achieved 26.42 and 0.7948, respectively. Furthermore, the method attained a stability score of 0.895, confirming its effectiveness in reducing temporal jitter.

Qualitative Analysis

Visual comparisons demonstrate that the proposed method produces sharper textures and more accurate geometric reconstructions. Unlike GAN-based approaches that often introduce wavy artifacts, this model maintains straight lines even at the extreme edges of the frame.

Qualitative comparison of the proposed method against state-of-the-art techniques on a dynamic scene.

Conclusion and Future Work

This research successfully demonstrates that spatiotemporal perception is critical for high-quality fisheye video rectification. By leveraging the relationship between motion and distortion, the proposed framework achieves superior accuracy and stability. Future work will focus on optimizing the computational efficiency of the 3D convolutions and attention mechanisms to enable real-time processing for applications such as live drone navigation and augmented reality.

Summary updated

Abstract

Although the distortion correction of fisheye images has been extensively studied, the correction of fisheye videos is still an elusive challenge. For different frames of the fisheye video, the existing image correction methods ignore the correlation of sequences, resulting in temporal jitter in the corrected video. To solve this problem, we propose a temporal weighting scheme to get a plausible global optical flow, which mitigates the jitter effect by progressively reducing the weight of frames. Subsequently, we observe that the inter-frame optical flow of the video is facilitated to perceive the local spatial deformation of the fisheye video. Therefore, we derive the spatial deformation through the flows of fisheye and distorted-free videos, thereby enhancing the local accuracy of the predicted result. However, the independent correction for each frame disrupts the temporal correlation. Due to the property of fisheye video, a distorted moving object may be able to find its distorted-free pattern at another moment. To this end, a temporal deformation aggregator is designed to reconstruct the deformation correlation between frames and provide a reliable global feature. Our method achieves an end-to-end correction and demonstrates superiority in correction quality and stability compared with the SOTA correction methods.
BibTeX
@inproceedings{yang2023spatiotemporal,
  author = {Shangrong Yang and Chunyu Lin and Kang Liao and Yao Zhao},
  title = {{Spatiotemporal Deformation Perception for Fisheye Video Rectification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25423},
}
Figure