Innovating Real Fisheye Image Correction with Dual Diffusion Architecture

Shangrong Yang, Chunyu Lin, Kang Liao, Yao Zhao

Abstract

Fisheye lenses provide expansive fields of view critical for autonomous driving and surveillance but introduce significant radial distortion. Traditional correction methods rely on synthetic datasets, creating a domain gap that hinders performance on real-world images. This paper introduces the Dual Diffusion Architecture (DDA), a novel framework that leverages the properties of Denoising Diffusion Probabilistic Models (DDPMs) to bridge this gap. By mapping both synthetic and unlabeled real-world images into a shared noise space, DDA achieves state-of-the-art rectification. Experimental results demonstrate a significant reduction in Fréchet Inception Distance (FID) from 65.8 to 24.9 on real-world datasets.

Introduction

Fisheye lenses are indispensable in applications requiring wide-angle perception. However, the resulting images suffer from severe non-linear distortion. While deep learning models can be trained on synthetic data—generated by applying mathematical distortion models to perspective images—they often fail to generalize to real-world fisheye photos. This failure stems from the "domain gap": real lenses deviate from ideal mathematical models, and real-world environmental conditions differ from synthetic textures.

The primary obstacle in training models for real-world correction is the lack of "ground-truth" paired data (i.e., a perfectly undistorted version of a real fisheye photo). Without these pairs, standard supervised learning techniques like CNNs or GANs struggle to learn the necessary transformations for real-world scenarios.

Methods

The proposed Dual Diffusion Architecture (DDA) addresses the domain gap through a multi-component system designed to learn from both labeled synthetic data and unlabeled real data.

Comparison of training paradigms: CNNs and GANs struggle with unlabeled real data, whereas the proposed DDPM-based DDA leverages noise-space equivalence.

Core Components

  1. Conditional Diffusion Module (CDM): Trained on paired synthetic datasets, the CDM learns the fundamental geometric rules of distortion. It uses the distorted image as a "condition" to guide the iterative generation of a rectified output.
  2. Unconditional Diffusion Module (UDM): The UDM is the key to handling real-world data. Since DDPMs learn by supervising noise rather than direct pixel values, the UDM can perceive the distribution of unlabeled real-world images. By treating noise as a "universal language," the model creates a bridge between synthetic and real domains.
  3. One-Pass Network (OPN): To mitigate the structural disparity between distorted circles and rectangular rectified images, the OPN acts as a pre-corrector. It predicts a "distortion flow" to roughly flatten the input, providing a more stable starting point for the diffusion process.

The DDA supports two operational modes: a high-speed "One-pass" mode for real-time applications and a high-quality "Inference" mode that utilizes the full iterative diffusion process.

Datasets

The model was evaluated using a combination of synthetic and real-world data:

  • Synthetic Dataset: Derived from the Places2 dataset, consisting of 44,000 perspective images distorted using a polynomial model to create ground-truth pairs.
  • Real-World Dataset: The Woodscape dataset, containing over 8,000 real fisheye images from automotive cameras. These images were used without undistorted labels during the training of the UDM.
A sample real-world fisheye image from the Woodscape dataset, exhibiting significant radial distortion.

Key Results and Insights

The DDA significantly outperforms existing state-of-the-art methods such as Blind, DCCNN, and PCN.

Quantitative Performance

On real-world benchmarks, the DDA's Inference mode achieved a FID of 24.9, a substantial improvement over the previous best of 65.8. This indicates that the corrected images are statistically much closer to real perspective photos. In terms of efficiency, the One-pass mode achieved a PSNR of 26.0, demonstrating high effectiveness even without iterative refinement.

Qualitative Advantages

Unlike previous methods that often introduce blur or over-rectify certain regions, DDA preserves sharp textures and maintains the integrity of straight lines across the entire field of view. The use of the OPN to "pre-flatten" images effectively reduces the artifacts typically associated with applying diffusion models to structurally dissimilar inputs.

Limitations and Future Work

While DDA sets a new benchmark, it faces challenges common to diffusion models, primarily the computational cost of iterative inference. Additionally, extreme radial distortion leads to inherent information loss at the edges of the fisheye circle, which remains difficult to recover perfectly. Future research may explore applying this "dual diffusion" strategy to other image-to-image translation tasks where real-world labels are difficult to obtain.

Summary updated

Abstract

Fisheye image rectification is hindered by synthetic models producing poor results for real-world correction. To address this, we propose a Dual Diffusion Architecture (DDA) for fisheye rectification that offers better practicality. The DDA leverages Denoising Diffusion Probabilistic Models (DDPMs) to gradually introduce bidirectional noise, allowing the synthesized and real images to develop into a consistent noise distribution. As a result, our network can perceive the distribution of unlabelled real fisheye images without relying on a transfer network, thus improving the performance of real fisheye correction. Additionally, we design an unsupervised one-pass network that generates a plausible new condition to strengthen guidance and address the non-negligible indeterminacy between the prior condition and the target. It can significantly affect the rectification task, especially in cases where radial distortion causes significant artifacts. This network can be regarded as an alternate scheme for fast producing reliable results without iterative inference. Compared to the state-of-the-art methods, our approach achieves superior performance in both synthetic and real fisheye image corrections.
BibTeX
@inproceedings{yang2023innovating,
  author = {Shangrong Yang and Chunyu Lin and Kang Liao and Yao Zhao},
  title = {{Innovating Real Fisheye Image Correction with Dual Diffusion Architecture}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01166},
}
Figure