Vision-Driven 2D Supervised Fine-Tuning Framework for Bird's Eye View Perception

Lei He, Qiaoyi Wang, Honglin Sun, Qing Xu, Bolin Gao, Shengbo Eben Li, Jianqiang Wang, Keqiang Li

Abstract

Bird’s Eye View (BEV) perception is the current standard for autonomous driving, yet training these models typically relies on expensive 3D LiDAR annotations. We present a novel Vision-Driven 2D Supervised Fine-Tuning Framework that enables BEV models to be refined using widely available 2D image annotations. By leveraging depth-guided matching to resolve spatial ambiguity, our approach improves model accuracy and generalization on camera-only vehicle fleets without the need for additional 3D ground truth data.

Introduction

Bird’s Eye View (BEV) perception has emerged as a critical technology for autonomous driving by mapping multi-view camera inputs into a unified 3D top-down perspective. However, the reliance on high-precision 3D ground truth (GT) labels—usually derived from expensive LiDAR sensors and complex manual annotation—presents a significant scalability bottleneck. Most production vehicles lack LiDAR, making it difficult to collect the necessary data to adapt models to new, unseen environments.

Comparison of conventional 3D-3D matching pipelines versus the proposed 3D-2D matching framework.

While 2D image annotation is cost-effective and mature, using 2D supervision to improve 3D perception is challenging due to depth ambiguity. A small object near the camera can appear identical to a large object further away on a 2D plane, making direct 2D-to-3D back-propagation noisy and unstable.

Methodology

The proposed framework facilitates fine-tuning by projecting 3D predictions back into the 2D image space and matching them with 2D labels.

The detailed architecture of the Vision-Driven 2D Supervised Fine-Tuning Framework, highlighting the depth-guided matching process.

1. Inference and Projection

The process begins with a pre-trained BEV model (such as BEVFormer) that generates 3D bounding box predictions. These boxes are then projected onto the 2D image planes of the vehicle's surround-view cameras.

2. Depth-Guided Matching

To address depth ambiguity, the framework incorporates an offline depth estimation model (DPT). This provides a depth map that acts as a spatial "anchor," ensuring that projected 3D boxes are associated with the correct 2D ground truth labels based on geometric consistency.

3. Optimization and Joint Training

The system employs the Hungarian algorithm for optimal bipartite matching between predictions and labels. The model is updated using a composite loss function comprising Focal Loss for classification, L1 Loss for bounding box regression, and GIoU Loss for overlap accuracy. To maintain existing 3D spatial knowledge and prevent catastrophic forgetting, a joint training strategy is used, mixing 2D-labeled fine-tuning data with original 3D-labeled datasets.

Experimental Results

The framework was validated on two industry-standard benchmarks: nuScenes and the Waymo Open Dataset.

  • nuScenes: The model achieved a Mean Average Precision (mAP) increase of 2.51 percentage points (from 0.2524 to 0.2775) and a 1.93 point improvement in the overall NuScenes Detection Score (NDS).
  • Waymo: The mAP improved by 1.21 percentage points.

Qualitative analysis indicates that the fine-tuned model is significantly more robust at detecting occluded objects and reducing false positives compared to the pre-trained baseline.

Conclusion

This research demonstrates that 2D image labels provide sufficient geometric constraints to refine 3D BEV models, provided the model already possesses a baseline understanding of 3D space. By bridging the gap between cheap 2D data and complex 3D perception, this framework offers a practical path for improving autonomous driving software across large-scale consumer vehicle fleets. Future work will explore extending this approach to local map construction and occupancy network perception.

Summary updated

BibTeX
@inproceedings{he2025vision,
  author = {Lei He and Qiaoyi Wang and Honglin Sun and Qing Xu and Bolin Gao and Shengbo Eben Li and Jianqiang Wang and Keqiang Li},
  title = {{Vision-Driven 2D Supervised Fine-Tuning Framework for Bird's Eye View Perception}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246559},
}
Figure