Introduction
Fisheye cameras are indispensable in autonomous driving and robotics due to their ultra-wide field of view (FoV). However, the circular projection of these lenses onto rectangular sensors inevitably leaves "blind zones"—unused black pixels at the corners of the image plane. Traditional computer vision pipelines treat image outpainting (filling in missing pixels) and semantic segmentation (labeling pixels) as isolated tasks. This paper introduces FishDreamer, a framework designed for the novel task of Fisheye Semantic Completion (FSC), which treats these two challenges as a unified, mutually beneficial process.
The Fisheye Semantic Completion Task
The core premise of FSC is that an object's visual texture and its semantic identity are intrinsically linked. For instance, observing the base of a building at the edge of a sensor's FoV provides a strong prior that the "blind" area above it contains more of the same structure. Conversely, knowing the semantic category of a region helps the model hallucinate more realistic textures. FishDreamer leverages this bidirectional relationship to generate semantically consistent scene completions beyond the physical limits of the camera sensor.
Methodology: The FishDreamer Architecture
FishDreamer employs a multi-stage architecture designed to handle the unique geometric distortions of fisheye imagery:
- Feature Extraction: A hierarchical Swin Transformer backbone extracts multi-scale features from the input fisheye image.
- Dual Heads: The model branches into an Outpainting Head (utilizing deconvolution layers for texture extrapolation) and a Segmentation Head (based on UPerNet for mask generation).
- Polar-aware Cross Attention (PCA): Standard attention mechanisms often fail to account for the radial nature of fisheye distortion. The PCA module introduces ring-shaped (annular) masks that constrain attention to specific polar distributions. This allows the segmentation and outpainting heads to exchange information in a way that respects the underlying lens geometry.
The framework is optimized using a comprehensive loss function suite, including adversarial (GAN) loss for visual realism, perceptual loss for structural integrity, and Lovász-softmax loss for precise segmentation.
Datasets and Benchmarking
A significant hurdle in FSC research is the lack of ground-truth data for the areas "outside" a standard fisheye FoV. To address this, the authors developed two new benchmarks: Cityscapes-BF and KITTI360-BF. These were created by projecting high-resolution pinhole images into a fisheye manifold, allowing the researchers to retain the original image data as ground truth for the "blind" regions.
Experimental Results
FishDreamer establishes a new state-of-the-art for unified outpainting and segmentation:
- Quantitative Performance: On the Cityscapes-BF dataset, the model achieved a mean Intersection over Union (mIoU) of 54.54% and a Peak Signal-to-Noise Ratio (PSNR) of 25.05 dB.
- Qualitative Superiority: Compared to existing outpainting methods like FisheyeEX, FishDreamer produces significantly sharper textures and more coherent semantic boundaries, avoiding the blurry artifacts common in previous approaches.
- Ablation Studies: The inclusion of the PCA module proved vital. Removing the polar-aware constraints led to a marked degradation in both reconstruction quality and segmentation accuracy, confirming that geometric priors are essential for fisheye-based tasks.
Conclusion and Future Directions
FishDreamer demonstrates that outpainting and segmentation are more effective when performed jointly, particularly in geometrically constrained domains like fisheye imaging. While the model shows impressive results on synthetic benchmarks, future work will focus on bridging the "synthetic-to-real" gap—accounting for real-world lens aberrations—and optimizing the Swin-based architecture for real-time performance on edge devices.
Summary updated