2D Supervised Monocular 3D Object Detection by Global-to-Local 3D Reconstruction

Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang ZhangarXiv 2023 · arXiv · OpenReview

Abstract

With the advent of the big model era, the demand for data has become more important. Especially in monocular 3D object detection, expensive manual annotations potentially limit further developments. Existing works have investigated weakly supervised algorithms with the help of LiDAR modality to generate 3D pseudo labels, which cannot be applied to ordinary videos. In this paper, we propose a novel paradigm, termed as BA$^2$-Det, leveraging the idea of global-to-local 3D reconstruction for 2D supervised monocular 3D object detection. Specifically, we recover 3D structures from monocular videos by scene-level global reconstruction with global bundle adjustment (BA) and obtain object clusters by the DoubleClustering algorithm. Learning from completely reconstructed objects in global BA, GBA-Learner predicts pseudo labels for occluded objects. Finally, we train an LBA-Learner with object-centric local BA to generalize the generated 3D pseudo labels to moving objects. Experiments on the large-scale Waymo Open Dataset show that the performance of BA$^2$-Det is on par with the fully-supervised BA-Det trained with 10% videos and even outperforms some pioneer fully-supervised methods. We also show the great potential of BA$^2$-Det for detecting open-set 3D objects in complex scenes.

Introduction

3D object detection is a fundamental component of robotic perception and autonomous driving. However, training robust models typically requires expensive LiDAR data and manual 3D cuboid annotations. While 2D labels are relatively cheap and abundant, bridging the "dimensionality gap" from 2D to 3D remains a significant challenge. Traditional Structure-from-Motion (SfM) techniques can reconstruct static scenes but struggle with moving objects and distant, occluded targets.

This paper introduces BA$^2$-Det, a framework that achieves high-performance 3D detection using only 2D bounding box annotations. By combining global scene reconstruction with a multi-stage generalization strategy, BA$^2$-Det effectively generates and refines 3D pseudo-labels from monocular video sequences.

Methodology

Pseudo Label Generation

The framework begins by generating a "seed" of 3D information. Using SfM, the system reconstructs the static environment from video. To extract individual object labels from the resulting point cloud, the authors developed a DoubleClustering algorithm:

  1. Local Point Clustering (LPC): Groups points within 2D boxes in individual frames.
  2. Global Point Clustering (GPC): Merges these groups across the entire video sequence to create clean, 3D object clusters.
  3. Box Fitting: Tight 3D cuboids are fitted to these clusters to create initial pseudo-labels.
Overview of the Pseudo Label Generation pipeline, showing the flow from video sequences to 3D box fitting via SfM and DoubleClustering.

Multi-Stage Generalization

Because SfM-based reconstruction is limited to static and well-captured objects, BA$^2$-Det employs a multi-stage strategy to generalize these labels to the entire scene:

  • Complete to Partial: A PointNet-like model ($G_\theta$) is trained to predict full 3D boxes even when input clusters are incomplete due to occlusion.
  • Static to Dynamic: This stage leverages the insight that 3D geometry is a class-level attribute. A monocular detector ($N_\theta$) trained on static pseudo-labels naturally generalizes to moving objects because the visual features (e.g., of a specific car model) remain consistent whether the object is stationary or in motion.
  • Close to Distant: For far-away objects with low pixel density, the system uses temporal aggregation and geometric optimization (Object-centric Local BA) to refine their 3D positions.
The multi-stage generalization process, illustrating the transition from static to dynamic objects and the optimization for distant targets.

Experiments and Results

The researchers evaluated BA$^2$-Det on the Waymo Open Dataset (WOD) and KITTI benchmarks. Key findings include:

  • Supervision Parity: On Waymo, BA$^2$-Det (using 0% 3D labels) performed on par with state-of-the-art detectors trained on 10% of the available 3D ground truth.
  • Pre-training Benefits: Using BA$^2$-Det for pre-training on unlabeled KITTI video resulted in a 20% relative performance improvement over standard baselines.
  • Open-Set Detection: By integrating masks from the Segment Anything Model (SAM), the system demonstrated the ability to detect "open-set" objects in 3D without prior 3D training data for those specific classes.

Conclusion

BA$^2$-Det presents a powerful paradigm for 2D-supervised 3D object detection. By utilizing global-to-local reconstruction and cross-modal generalization, it significantly reduces the reliance on expensive 3D annotations. While currently dependent on camera ego-motion for initial SfM reconstruction, the framework's ability to generalize from static to dynamic objects marks a significant step toward scalable, weakly supervised perception systems.

Summary updated

Abstract

With the advent of the big model era, the demand for data has become more important. Especially in monocular 3D object detection, expensive manual annotations potentially limit further developments. Existing works have investigated weakly supervised algorithms with the help of LiDAR modality to generate 3D pseudo labels, which cannot be applied to ordinary videos. In this paper, we propose a novel paradigm, termed as BA$^2$-Det, leveraging the idea of global-to-local 3D reconstruction for 2D supervised monocular 3D object detection. Specifically, we recover 3D structures from monocular videos by scene-level global reconstruction with global bundle adjustment (BA) and obtain object clusters by the DoubleClustering algorithm. Learning from completely reconstructed objects in global BA, GBA-Learner predicts pseudo labels for occluded objects. Finally, we train an LBA-Learner with object-centric local BA to generalize the generated 3D pseudo labels to moving objects. Experiments on the large-scale Waymo Open Dataset show that the performance of BA$^2$-Det is on par with the fully-supervised BA-Det trained with 10% videos and even outperforms some pioneer fully-supervised methods. We also show the great potential of BA$^2$-Det for detecting open-set 3D objects in complex scenes. The code will be made available. Project page: https://ba2det.site .
BibTeX
@misc{he2023d,
  author = {Jiawei He and Yuqi Wang and Yuntao Chen and Zhaoxiang Zhang},
  title = {{2D Supervised Monocular 3D Object Detection by Global-to-Local 3D Reconstruction}},
  year = {2023},
  eprint = {2306.05418},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.05418},
}
Figure