FishDreamer: Towards Fisheye Semantic Completion via Unified Image Outpainting and Segmentation

Hao Shi, Yu Li, Kailun Yang, Jiaming Zhang, Kunyu Peng, Alina Roitberg, Yaozu Ye, Huajian Ni, Kaiwei Wang, Rainer Stiefelhagen

Introduction

Fisheye cameras are indispensable in autonomous driving and robotics due to their ultra-wide field of view (FoV). However, the circular projection of these lenses onto rectangular sensors inevitably leaves "blind zones"—unused black pixels at the corners of the image plane. Traditional computer vision pipelines treat image outpainting (filling in missing pixels) and semantic segmentation (labeling pixels) as isolated tasks. This paper introduces FishDreamer, a framework designed for the novel task of Fisheye Semantic Completion (FSC), which treats these two challenges as a unified, mutually beneficial process.

The Fisheye Semantic Completion Task

The core premise of FSC is that an object's visual texture and its semantic identity are intrinsically linked. For instance, observing the base of a building at the edge of a sensor's FoV provides a strong prior that the "blind" area above it contains more of the same structure. Conversely, knowing the semantic category of a region helps the model hallucinate more realistic textures. FishDreamer leverages this bidirectional relationship to generate semantically consistent scene completions beyond the physical limits of the camera sensor.

Methodology: The FishDreamer Architecture

FishDreamer employs a multi-stage architecture designed to handle the unique geometric distortions of fisheye imagery:

  1. Feature Extraction: A hierarchical Swin Transformer backbone extracts multi-scale features from the input fisheye image.
  2. Dual Heads: The model branches into an Outpainting Head (utilizing deconvolution layers for texture extrapolation) and a Segmentation Head (based on UPerNet for mask generation).
  3. Polar-aware Cross Attention (PCA): Standard attention mechanisms often fail to account for the radial nature of fisheye distortion. The PCA module introduces ring-shaped (annular) masks that constrain attention to specific polar distributions. This allows the segmentation and outpainting heads to exchange information in a way that respects the underlying lens geometry.

The framework is optimized using a comprehensive loss function suite, including adversarial (GAN) loss for visual realism, perceptual loss for structural integrity, and Lovász-softmax loss for precise segmentation.

Datasets and Benchmarking

A significant hurdle in FSC research is the lack of ground-truth data for the areas "outside" a standard fisheye FoV. To address this, the authors developed two new benchmarks: Cityscapes-BF and KITTI360-BF. These were created by projecting high-resolution pinhole images into a fisheye manifold, allowing the researchers to retain the original image data as ground truth for the "blind" regions.

Experimental Results

FishDreamer establishes a new state-of-the-art for unified outpainting and segmentation:

  • Quantitative Performance: On the Cityscapes-BF dataset, the model achieved a mean Intersection over Union (mIoU) of 54.54% and a Peak Signal-to-Noise Ratio (PSNR) of 25.05 dB.
  • Qualitative Superiority: Compared to existing outpainting methods like FisheyeEX, FishDreamer produces significantly sharper textures and more coherent semantic boundaries, avoiding the blurry artifacts common in previous approaches.
  • Ablation Studies: The inclusion of the PCA module proved vital. Removing the polar-aware constraints led to a marked degradation in both reconstruction quality and segmentation accuracy, confirming that geometric priors are essential for fisheye-based tasks.

Conclusion and Future Directions

FishDreamer demonstrates that outpainting and segmentation are more effective when performed jointly, particularly in geometrically constrained domains like fisheye imaging. While the model shows impressive results on synthetic benchmarks, future work will focus on bridging the "synthetic-to-real" gap—accounting for real-world lens aberrations—and optimizing the Swin-based architecture for real-time performance on edge devices.

Summary updated

Abstract

This paper raises the new task of Fisheye Semantic Completion (FSC), where dense texture, structure, and semantics of a fisheye image are inferred even beyond the sensor field-of-view (FoV). Fisheye cameras have larger FoV than ordinary pinhole cameras, yet its unique special imaging model naturally leads to a blind area at the edge of the image plane. This is suboptimal for safety-critical applications since important perception tasks, such as semantic segmentation, become very challenging within the blind zone. Previous works considered the out-FoV outpainting and in-FoV segmentation separately. However, we observe that these two tasks are actually closely coupled. To jointly estimate the tightly intertwined complete fisheye image and scene semantics, we introduce the new FishDreamer which relies on successful ViTs enhanced with a novel Polar-aware Cross Attention module (PCA) to leverage dense context and guide semantically-consistent content generation while considering different polar distributions. In addition to the contribution of the novel task and architecture, we also derive Cityscapes-BF and KITTI360-BF datasets to facilitate training and evaluation of this new track. Our experiments demonstrate that the proposed FishDreamer outperforms methods solving each task in isolation and surpasses alternative approaches on the Fisheye Semantic Completion. Code and datasets are publicly available at FishDreamer.
BibTeX
@inproceedings{shi2023fishdreamer,
  author = {Hao Shi and Yu Li and Kailun Yang and Jiaming Zhang and Kunyu Peng and Alina Roitberg and Yaozu Ye and Huajian Ni and Kaiwei Wang and Rainer Stiefelhagen},
  title = {{FishDreamer: Towards Fisheye Semantic Completion via Unified Image Outpainting and Segmentation}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  doi = {10.1109/cvprw59228.2023.00684},
}
Figure