F2BEV: Bird's Eye View Generation from Surround-View Fisheye Camera Images for Automated Driving

Ekta U. Samani, Tao Feng, Harshavardhan R. Dasari, Sihao Ding, Ashis G. BanerjeeIROS 2023

Abstract

Bird's Eye View (BEV) representations are tremendously useful for perception-related automated driving tasks. However, generating BEVs from surround-view fisheye camera images is challenging due to the strong distortions introduced by such wide-angle lenses. We take the first step in addressing this challenge and introduce a baseline, F2BEV, to generate discretized BEV height maps and BEV semantic segmentation maps from fisheye images. F2BEV consists of a distortion-aware spatial cross attention module for querying and consolidating spatial information from fisheye image features in a transformer-style architecture followed by a task-specific head. We evaluate single-task and multi-task variants of F2BEV on our synthetic FB-SSEM dataset, all of which generate better BEV height and segmentation maps (in terms of the IoU) than a state-of-the-art BEV generation method operating on undistorted fisheye images. We also demonstrate discretized height map generation from real-world fisheye images using F2BEV. Our dataset is publicly available at https://github.com/volvo-cars/FB-SSEM-dataset

Introduction

Automated driving systems require a comprehensive 360-degree understanding of their environment. Surround-view systems often utilize four wide-angle fisheye cameras to achieve this coverage with a minimal sensor footprint. However, the extreme radial distortion inherent in fisheye lenses—where straight lines appear curved and peripheral objects are heavily warped—presents a significant challenge for standard computer vision pipelines designed for pinhole camera models.

Traditionally, engineers address this by "undistorting" images into perspective views. This pre-processing step is suboptimal as it typically requires cropping the edges or introducing interpolation artifacts, effectively discarding the wide-angle data that makes fisheye cameras valuable. F2BEV (Fisheye to BEV) is the first framework designed to bypass this step, learning directly from raw, distorted surround-view images to generate unified BEV representations.

Methodology

F2BEV utilizes a transformer-based architecture to transform perspective-view features into a unified BEV grid. The core of the system is the Distortion-Aware Spatial Cross-Attention (DA-SCA) module. Unlike standard attention mechanisms that assume linear projections, DA-SCA integrates a unified camera model that accounts for both radial and tangential distortion. For every query point in the BEV grid, the system calculates the precise pixel coordinates in the distorted fisheye image that correspond to that 3D location.

The F2BEV architecture featuring the Distortion-Aware Spatial Cross-Attention (DA-SCA) module.

To enhance depth perception and scene understanding, the model incorporates Temporal Self-Attention, using ego-motion data to align and integrate features from previous timesteps. The architecture concludes with task-specific heads designed to produce two primary outputs:

  1. Discretized Height Maps: Classifying regions into bins such as "below car level," "at car level," or "above car level."
  2. Semantic Segmentation: Identifying environmental categories including vehicles, buses, EV chargers, and driveable surfaces.

Experimental Setup

The researchers evaluated F2BEV using two primary data sources:

  • FB-SSEM (Fisheye BEV - Simulated Sequences of Ego Motion): A synthetic dataset created in the Unity engine containing 20,000 samples across 20 sequences. It provides four fisheye views, ego-motion data, and ground-truth BEV maps.
  • Real-World Data: Testing was conducted on footage from a Volvo XC90 equipped with four 210° Field of View (FoV) fisheye cameras to demonstrate the model's practical versatility.

Key Results

F2BEV was benchmarked against a state-of-the-art baseline, BEVFormer, which operated on traditionally undistorted images. F2BEV demonstrated superior performance across all metrics:

  • Height Map Generation: F2BEV achieved a Mean Intersection over Union (mIoU) of 0.9106, significantly outperforming the undistorted baseline's 0.8612.
  • Semantic Segmentation: The model showed an approximate 9% improvement in frequency-weighted IoU over the baseline.
Qualitative results of F2BEV semantic segmentation on the synthetic FB-SSEM dataset.

Qualitative analysis revealed that F2BEV effectively preserves the periphery of the sensor's FoV, detecting objects that would have been lost during standard undistortion. Furthermore, by sampling the image at multiple "height anchors" for the same BEV coordinate, the model implicitly learns to infer object height and distance without requiring LiDAR.

Discussion and Future Work

While F2BEV establishes a strong baseline, several areas for improvement remain. A notable challenge is occlusion mismatch, where the model is penalized for failing to detect objects hidden behind large obstacles (like buses) that are visible in the "god-view" ground truth but physically invisible to the vehicle's cameras. Additionally, the model currently relies on precise manual camera calibration. Future iterations could explore calibration-independent modules or continuous height estimation to move beyond discretized bins.

Conclusion

F2BEV demonstrates that neural networks can effectively "see through" extreme lens distortion when the underlying camera physics are integrated into the attention mechanism. By preserving the full field of view of fisheye sensors, F2BEV provides a more robust and accurate BEV representation for automated driving tasks than traditional undistortion-based methods.

Summary updated

Abstract

Bird's Eye View (BEV) representations are tremendously useful for perception-related automated driving tasks. However, generating BEVs from surround-view fisheye camera images is challenging due to the strong distortions introduced by such wide-angle lenses. We take the first step in addressing this challenge and introduce a baseline, F2BEV, to generate discretized BEV height maps and BEV semantic segmentation maps from fisheye images. F2BEV consists of a distortion-aware spatial cross attention module for querying and consolidating spatial information from fisheye image features in a transformer-style architecture followed by a task-specific head. We evaluate single-task and multi-task variants of F2BEV on our synthetic FB-SSEM dataset, all of which generate better BEV height and segmentation maps (in terms of the IoU) than a state-of-the-art BEV generation method operating on undistorted fisheye images. We also demonstrate discretized height map generation from real-world fisheye images using F2BEV. Our dataset is publicly available at https://github.com/volvo-cars/FB-SSEM-dataset
BibTeX
@inproceedings{samani2023f,
  author = {Ekta U. Samani and Tao Feng and Harshavardhan R. Dasari and Sihao Ding and Ashis G. Banerjee},
  title = {{F2BEV: Bird's Eye View Generation from Surround-View Fisheye Camera Images for Automated Driving}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
}
Figure