FishBEV: Distortion-Resilient Bird's Eye View Segmentation with Surround-View Fisheye Cameras

Hang Li, Dianmo Sheng, Qiankun Dong, Zichun Wang, Zhiwei Xu, Tao LiarXiv 2025 · DOI · arXiv

Abstract

Bird’s Eye View (BEV) segmentation is a cornerstone of autonomous driving perception, providing a unified top-down representation for downstream tasks. While pinhole camera-based BEV methods have matured, surround-view fisheye cameras present unique challenges due to extreme radial distortion and non-uniform resolution. This paper introduces FishBEV, a framework specifically engineered for fisheye-based BEV segmentation. FishBEV leverages a foundation-model-based backbone for distortion-resilient feature extraction, an uncertainty-aware spatial cross-attention mechanism for multi-view fusion, and a distance-aware temporal self-attention module for stable long-range perception. Evaluated on the Synwoodscapes dataset, FishBEV achieves a state-of-the-art mIoU of 64.22%, significantly outperforming existing methods.

Introduction

Autonomous vehicles require a comprehensive 360-degree understanding of their environment. Surround-view fisheye cameras are increasingly popular due to their ultra-wide field of view (FOV), often exceeding 180 degrees per lens, which allows for full coverage with fewer sensors. However, the transition from pinhole to fisheye cameras in BEV segmentation is non-trivial.

Fisheye lenses introduce severe geometric distortions where straight lines appear curved, and resolution decreases significantly towards the image periphery. Furthermore, the large overlap between adjacent cameras creates ambiguity in multi-view correspondence, making it difficult to determine which sensor provides the most reliable data for a given spatial coordinate on the top-down map. FishBEV is designed to address these specific geometric and temporal inconsistencies.

Comparison of Field of View (FOV) coverage between standard pinhole and surround-view fisheye camera configurations.

Methodology

FishBEV addresses the inherent limitations of fisheye optics through three novel modules designed to bridge the gap between distorted image space and a clean BEV representation.

Distortion-Resilient Multi-scale Extraction (DRME)

Traditional backbones like ResNet, typically trained on rectilinear images, struggle to generalize to the non-linear distortions of fisheye lenses. FishBEV employs DINOv2, a self-supervised foundation model, as its backbone. DINOv2’s features exhibit superior robustness to geometric transformations compared to standard supervised backbones. This is paired with a Feature Pyramid Network (FPN) to maintain high-resolution details necessary for segmenting small or distant objects.

Uncertainty-aware Spatial Cross-Attention (U-SCA)

To handle overlapping views and varying distortion levels, U-SCA treats feature reliability as a learnable aleatoric uncertainty. By predicting the mean and variance for each camera's input, the model can perform Monte Carlo sampling to weight contributions. This allows the network to dynamically "trust" high-confidence regions—typically the center of a lens—over the heavily distorted and low-confidence edges of an adjacent camera.

Intensity map representing feature reliability or uncertainty across the field of view, used to guide spatial fusion.

Distance-aware Temporal Self-Attention (D-TSA)

Temporal consistency is maintained using a distance-gated mechanism. Near-field objects, which move rapidly across the FOV, rely primarily on the current frame for high-precision localization. Conversely, far-field objects, which are more distorted and move slowly, benefit from historical context to improve stability and mitigate noise. This distance-dependent approach ensures that the temporal fusion strategy matches the physical dynamics of the scene.

Experimental Results

The model was benchmarked on Synwoodscapes, a synthetic dataset featuring four 190° FOV fisheye cameras simulated in CARLA. FishBEV-Large achieved 64.22% mIoU, surpassing the previous state-of-the-art fisheye method, F2BEV, by 10.83% and standard BEVFormer by nearly 15%.

Significant gains were observed in challenging dynamic categories such as "Four-wheeler-vehicle" (+20.5% over F2BEV) and "Sidewalk" (+19.6%). Notably, even the parameter-efficient FishBEV-Small (12.6M parameters) outperformed all larger baseline models, demonstrating that architectural specialization for fisheye geometry is more effective than simply increasing model scale.

Conclusion

FishBEV demonstrates that specialized architectural considerations—robust backbones, uncertainty modeling, and distance-aware temporal fusion—are essential for high-performance fisheye BEV perception. By explicitly addressing the geometric and temporal characteristics of fisheye lenses, FishBEV sets a new benchmark for surround-view semantic mapping. Future work will focus on extending this framework to 3D object detection and addressing the domain gap between synthetic and real-world fisheye data.

Summary updated

BibTeX
@misc{li2025fishbev,
  author = {Hang Li and Dianmo Sheng and Qiankun Dong and Zichun Wang and Zhiwei Xu and Tao Li},
  title = {{FishBEV: Distortion-Resilient Bird's Eye View Segmentation with Surround-View Fisheye Cameras}},
  year = {2025},
  eprint = {2509.13681},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.13681},
}
Figure