Abstract
Fisheye lenses provide expansive fields of view critical for autonomous driving and surveillance but introduce significant radial distortion. Traditional correction methods rely on synthetic datasets, creating a domain gap that hinders performance on real-world images. This paper introduces the Dual Diffusion Architecture (DDA), a novel framework that leverages the properties of Denoising Diffusion Probabilistic Models (DDPMs) to bridge this gap. By mapping both synthetic and unlabeled real-world images into a shared noise space, DDA achieves state-of-the-art rectification. Experimental results demonstrate a significant reduction in Fréchet Inception Distance (FID) from 65.8 to 24.9 on real-world datasets.
Introduction
Fisheye lenses are indispensable in applications requiring wide-angle perception. However, the resulting images suffer from severe non-linear distortion. While deep learning models can be trained on synthetic data—generated by applying mathematical distortion models to perspective images—they often fail to generalize to real-world fisheye photos. This failure stems from the "domain gap": real lenses deviate from ideal mathematical models, and real-world environmental conditions differ from synthetic textures.
The primary obstacle in training models for real-world correction is the lack of "ground-truth" paired data (i.e., a perfectly undistorted version of a real fisheye photo). Without these pairs, standard supervised learning techniques like CNNs or GANs struggle to learn the necessary transformations for real-world scenarios.
Methods
The proposed Dual Diffusion Architecture (DDA) addresses the domain gap through a multi-component system designed to learn from both labeled synthetic data and unlabeled real data.
Core Components
- Conditional Diffusion Module (CDM): Trained on paired synthetic datasets, the CDM learns the fundamental geometric rules of distortion. It uses the distorted image as a "condition" to guide the iterative generation of a rectified output.
- Unconditional Diffusion Module (UDM): The UDM is the key to handling real-world data. Since DDPMs learn by supervising noise rather than direct pixel values, the UDM can perceive the distribution of unlabeled real-world images. By treating noise as a "universal language," the model creates a bridge between synthetic and real domains.
- One-Pass Network (OPN): To mitigate the structural disparity between distorted circles and rectangular rectified images, the OPN acts as a pre-corrector. It predicts a "distortion flow" to roughly flatten the input, providing a more stable starting point for the diffusion process.
The DDA supports two operational modes: a high-speed "One-pass" mode for real-time applications and a high-quality "Inference" mode that utilizes the full iterative diffusion process.
Datasets
The model was evaluated using a combination of synthetic and real-world data:
- Synthetic Dataset: Derived from the Places2 dataset, consisting of 44,000 perspective images distorted using a polynomial model to create ground-truth pairs.
- Real-World Dataset: The Woodscape dataset, containing over 8,000 real fisheye images from automotive cameras. These images were used without undistorted labels during the training of the UDM.
Key Results and Insights
The DDA significantly outperforms existing state-of-the-art methods such as Blind, DCCNN, and PCN.
Quantitative Performance
On real-world benchmarks, the DDA's Inference mode achieved a FID of 24.9, a substantial improvement over the previous best of 65.8. This indicates that the corrected images are statistically much closer to real perspective photos. In terms of efficiency, the One-pass mode achieved a PSNR of 26.0, demonstrating high effectiveness even without iterative refinement.
Qualitative Advantages
Unlike previous methods that often introduce blur or over-rectify certain regions, DDA preserves sharp textures and maintains the integrity of straight lines across the entire field of view. The use of the OPN to "pre-flatten" images effectively reduces the artifacts typically associated with applying diffusion models to structurally dissimilar inputs.
Limitations and Future Work
While DDA sets a new benchmark, it faces challenges common to diffusion models, primarily the computational cost of iterative inference. Additionally, extreme radial distortion leads to inherent information loss at the edges of the fisheye circle, which remains difficult to recover perfectly. Future research may explore applying this "dual diffusion" strategy to other image-to-image translation tasks where real-world labels are difficult to obtain.
Summary updated