SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images

Nikhil Gosala, Kürsat Petek, Paulo L. J. Drews-Jr, Wolfram Burgard, Abhinav Valada

Introduction: The Challenge of 3D Perception

In autonomous driving, Bird’s-Eye-View (BEV) maps are critical for path planning and obstacle avoidance. However, generating these top-down perspectives from a standard monocular frontal camera typically requires massive amounts of 3D ground truth data, such as LiDAR scans and hand-annotated 3D bounding boxes. These labels are prohibitively expensive to produce at scale.

The SkyEye framework leverages self-supervision from frontal view video to predict BEV maps without 3D labels.

SkyEye introduces a self-supervised framework that bypasses the need for 3D annotations. By utilizing temporal consistency in video and automatically generated pseudolabels, it achieves performance comparable to fully supervised models using only 2D semantic labels and ego-motion data.

Methodology: Lifting 2D to 3D

The core of SkyEye is a dual-mode self-supervision strategy integrated into an architecture that "lifts" 2D image features into a 3D voxel representation.

Detailed architecture of SkyEye, including the Image Encoder, Lifting Module, and the dual supervision paths (FV Head for temporal consistency and BEV Head for pseudolabel training).

1. Architecture and the Voxel Bridge

SkyEye employs an EfficientDet-D3 backbone to extract features from a frontal image. These features are projected into a 3D voxel grid using a learned depth distribution. This voxel grid acts as a unified representation that can be rendered back into the Frontal View (FV) for supervision or flattened into a BEV map for prediction.

2. Implicit Supervision via Temporal Consistency

To learn the 3D structure without explicit labels, the model uses temporal consistency. Given the current frame's voxel grid and the vehicle's ego-motion, the model predicts the appearance of future video frames. If the internal 3D representation is accurate, the re-projected scene should match the actual future camera observations.

3. Explicit Supervision via Pseudolabeling

To provide a more direct training signal, SkyEye generates "silver standard" BEV labels. It combines a self-supervised depth network with existing 2D semantic labels to project points into 3D:

  • "Stuff" Densification: Static elements like roads are densified to create continuous surfaces.
  • "Thing" Bounding Box Fitting: Dynamic objects (e.g., cars) are clustered using DBSCAN, and geometric shapes (like ellipses) are fitted using RANSAC to create dense top-down targets.

Datasets and Evaluation

The model was trained and evaluated on the KITTI-360 dataset. To assess cross-dataset generalization, it was also tested on the Waymo Open Dataset. The researchers have released their generated BEV semantic datasets for both KITTI-360 and Waymo to support further research in self-supervised mapping.

Key Results and Insights

  • Performance Parity: SkyEye achieved a mean Intersection-over-Union (mIoU) of 30.49% on KITTI-360, matching the 30.42% mIoU of the state-of-the-art fully supervised model, PoBEV.
  • Robust Generalization: Unlike supervised models that often overfit to specific dataset labeling styles, SkyEye’s reliance on geometric principles allowed it to generalize significantly better to the Waymo dataset.
  • Efficiency: With only 14.6 million parameters and a runtime of ~77.84 ms per sample, the model is optimized for real-time applications.
  • Data Efficiency: In ablation studies, SkyEye remained competitive even when using only 1% of the available supervision, demonstrating the robustness of its self-supervised signals.

Limitations and Future Outlook

While SkyEye excels in static environments, highly dynamic scenes remain a challenge. Independent motion from other vehicles can cause "stretching" artifacts in the predicted maps because the implicit supervision assumes a static world warped by ego-motion. Future work aims to incorporate dedicated temporal baselines to better track moving targets and potentially remove the requirement for 2D frontal labels by using generic pre-trained segmentation networks.

Summary updated

Abstract

Bird's-Eye-View (BEV) semantic maps have become an essential component of automated driving pipelines due to the rich representation they provide for decision-making tasks. However, existing approaches for generating these maps still follow a fully supervised training paradigm and hence rely on large amounts of annotated BEV data. In this work, we address this limitation by proposing the first self-supervised approach for generating a BEV semantic map using a single monocular image from the frontal view (FV). During training, we overcome the need for BEV ground truth annotations by leveraging the more easily available FV semantic annotations of video sequences. Thus, we propose the SkyEye architecture that learns based on two modes of self-supervision, namely, implicit supervision and explicit supervision. Implicit supervision trains the model by enforcing spatial consistency of the scene over time based on FV semantic sequences, while explicit supervision exploits BEV pseudolabels generated from FV semantic annotations and self-supervised depth estimates. Extensive evaluations on the KITTI-360 dataset demonstrate that our self-supervised approach performs on par with the state-of-the-art fully supervised methods and achieves competitive results using only 1% of direct supervision in the BEV compared to fully supervised approaches. Finally, we publicly release both our code and the BEV datasets generated from the KITTI-360 and Waymo datasets.
BibTeX
@inproceedings{gosala2023skyeye,
  author = {Nikhil Gosala and Kürsat Petek and Paulo L. J. Drews-Jr and Wolfram Burgard and Abhinav Valada},
  title = {{SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01431},
}
Figure