Universal Semantic Segmentation for Fisheye Urban Driving Images

Yaozu Ye, Kailun Yang, Kaite Xiang, Juan Wang, Kaiwei Wang

Introduction

Fisheye cameras are indispensable for autonomous driving, offering a wide field of view (FoV) that eliminates critical blind spots. However, their extreme radial distortion presents a significant challenge for standard semantic segmentation models, which are typically optimized for rectilinear (pinhole) images. Furthermore, the scarcity of high-quality, pixel-annotated fisheye datasets hinders the development of robust perception systems. To address these issues, this research proposes a seven degrees of freedom (7-DoF) data augmentation method that synthetically generates fisheye images from standard rectilinear datasets during training.

The 7-DoF Augmentation Framework

The core innovation lies in bridging the "distortion gap" by simulating various camera poses and lens characteristics. Unlike standard augmentations that may destroy radial patterns, the 7-DoF method maintains the mathematical integrity of the fisheye projection.

The geometric projection model used to transform rectilinear images into synthetic fisheye data across world, camera, and image coordinate systems.

The augmentation pipeline incorporates:

  • 6-DoF Pose Simulation: This accounts for the camera's rotation (pitch, yaw, roll) and translation (X, Y, Z), mimicking different mounting positions on a vehicle.
  • 1-DoF Focal Length ($f$): By varying the focal length in the projection model ($r = f\theta$), the system can simulate a wide array of lens types and distortion intensities.

This approach allows a model trained on a dataset like Cityscapes to generalize to any fisheye camera configuration without requiring manual re-annotation.

Experimental Results and Robustness

The researchers utilized SwiftNet-18, an efficient CNN backbone suitable for real-time automotive applications. The 7-DoF augmentation was tested against various baselines on synthetic test sets with varying focal lengths.

An example of a synthetically generated fisheye image used to train the model, demonstrating the realistic radial distortion applied to a standard urban driving scene.

Key findings include:

  • Impact of Focal Length: Ablation studies revealed that varying the focal length (RandF) during training provided the most significant boost in Mean Intersection over Union (mIoU), as it teaches the model to be invariant to specific lens "stretch."
  • Quantitative Performance: On highly distorted test sets ($f=200$), the 7-DoF model achieved an mIoU of 0.6093, compared to 0.5562 for the baseline.
  • Universal Applicability: The model demonstrated high robustness across different lens types and mounting heights. Qualitative testing on real-world fisheye footage from smartphones confirmed the model's ability to accurately segment roads, vehicles, and pedestrians in diverse environments.

Conclusion

By leveraging a mathematically grounded 7-DoF augmentation strategy, this work enables universal semantic segmentation for fisheye cameras using only existing rectilinear datasets. This significantly reduces the data bottleneck in autonomous driving perception. Future work will explore CNN architectures specifically designed for non-linear geometries, such as those incorporating deformable convolutions, to further bridge the domain gap between synthetic and real-world optical effects.

Summary updated

BibTeX
@inproceedings{ye2020universal,
  author = {Yaozu Ye and Kailun Yang and Kaite Xiang and Juan Wang and Kaiwei Wang},
  title = {{Universal Semantic Segmentation for Fisheye Urban Driving Images}},
  booktitle = {SMC},
  year = {2020},
  doi = {10.1109/smc42975.2020.9283099},
}
Figure