Abstract
Visual bird's eye view (BEV) semantic segmentation helps autonomous vehicles understand the surrounding environment only from images, including static elements (e.g., roads) and dynamic elements (e.g., vehicles, pedestrians). However, the high cost of annotation procedures of full-supervised methods limits the capability of the visual BEV semantic segmentation, which usually needs HD maps, 3D object bounding boxes, and camera extrinsic matrixes. In this paper, we present a novel semi-supervised framework for visual BEV semantic segmentation to boost performance by exploiting unlabeled images during the training. A consistency loss that makes full use of unlabeled data is then proposed to constrain the model on not only semantic prediction but also the BEV feature. Furthermore, we propose a novel and effective data augmentation method named conjoint rotation which reasonably augments the dataset while maintaining the geometric relationship between the front-view images and the BEV semantic segmentation. Extensive experiments on the nuScenes and Argoverse datasets show that our semi-supervised framework can effectively improve prediction accuracy. To the best of our knowledge, this is the first work that explores improving visual BEV semantic segmentation performance using unlabeled data. The code is available at https://github.com/Junyu-Z/Semi-BEVseg
Introduction
Autonomous vehicles require a top-down, "Bird’s Eye View" (BEV) understanding of their environment for effective path planning and obstacle avoidance. While generating these maps from front-view (FV) cameras is cost-effective, training the underlying models is traditionally data-intensive. Standard supervised learning requires massive amounts of expensive ground truth data, including HD maps and 3D bounding boxes.
While raw driving video is abundant, labeling it for BEV is difficult and costly. This paper introduces the first semi-supervised learning (SSL) framework for visual BEV semantic segmentation. By leveraging unlabeled images through a Mean Teacher architecture and a novel "conjoint rotation" data augmentation that preserves camera-to-ground geometry, we achieve significant performance gains compared to supervised-only baselines.
Methodology
The proposed framework utilizes a Mean Teacher architecture, consisting of a Student network (updated via gradient descent) and a Teacher network (an exponential moving average of the Student).
Dual Consistency Loss
To effectively utilize unlabeled data, the framework enforces consistency on two levels:
- Segmentation Consistency ($L_{sc}$): Ensuring the predicted maps for an image and its flipped version match.
- Feature Consistency ($L_{fc}$): Ensuring the internal BEV features (before the final prediction) remain spatially consistent under perturbations.
Conjoint Rotation Augmentation
Standard 2D augmentations, such as random cropping, often break the complex geometric relationship between the FV image and the BEV plane. We propose Conjoint Rotation, which mathematically links a rotation of the camera view with a corresponding rotation of the ground plane using homography transformations. This allows the model to learn from diverse views without losing the spatial logic required to project pixels from 2D to 3D.
Experiments and Results
We evaluated our framework on the nuScenes dataset, artificially limiting the labeled portion to 5%, 10%, 20%, and 40% to test the SSL framework's effectiveness.
Quantitative Performance
Our semi-supervised approach consistently outperforms supervised-only baselines. With only 10% labeled data, the proposed method achieved a 20.1% mIoU, significantly outperforming the supervised-only baseline of 17.3%. Notably, the model trained on just 10% labeled data (plus unlabeled data) outperformed a fully supervised model trained on 20% labeled data, effectively doubling the value of each labeled sample.
Geometry and SSL
Ablation studies showed that Conjoint Rotation alone provided a massive boost (approx. 2-3% mIoU) even before adding SSL consistency losses. Furthermore, we observed that standard 2D SSL methods (like CPS or UniMatch) often underperform in this context, suggesting that BEV tasks require specialized strategies that respect 3D spatial constraints.
Conclusion
This work demonstrates that unlabeled data can significantly boost the performance of visual BEV semantic segmentation. By maintaining geometric consistency through conjoint rotation and enforcing dual-level consistency losses, our framework reduces the reliance on expensive annotations. Future work will explore extending this framework to 3D object detection and semantic occupancy tasks.
Summary updated