Prompt3D: Random Prompt Assisted Weakly-Supervised 3D Object Detection

Xiaohong Zhang, Huisheng Ye, Jingwen Li, Qinyu Tang, Yuanqi Li, Yanwen Guo, Jie GuoCVPR 2024 · OpenReview

Abstract

The prohibitive cost of annotations for fully supervised 3D indoor object detection limits its practicality. In this work, we propose Random Prompt Assisted Weakly-supervised 3D Object Detection, termed as Prompt3D, a weakly-supervised approach that leverages position-level labels to overcome this challenge. Explicitly, our method focuses on enhancing labeling using synthetic scenes crafted from 3D shapes generated via random prompts. First, a Synthetic Scene Generation (SSG) module is introduced to assemble synthetic scenes with a curated collection of 3D shapes, created via random prompts for each category. These scenes are enriched with automatically generated point-level annotations, providing a robust supervisory framework for training the detection algorithm. To enhance the transfer of knowledge from virtual to real datasets, we then introduce a Prototypical Proposal Feature Alignment (PPFA) module. This module effectively alleviates the domain gap by directly minimizing the distance between feature prototypes of the same class proposals across two domains. Compared with sota BR, our method improves by 5.4% and 8.7% on mAP with VoteNet and GroupFree3D serving as detectors respectively, demonstrating the effectiveness of our proposed method.

Introduction

3D object detection—identifying the location, size, and orientation of objects in a 3D point cloud—is a cornerstone of robotics and augmented reality. However, training these models typically requires "strong supervision" in the form of thousands of manually drawn 3D bounding boxes, which is incredibly labor-intensive and expensive.

"Weakly-supervised" methods attempt to train models using simpler labels, like a single point at the center of an object. While much cheaper to produce, these labels lack the geometric detail (size and shape) the model needs to learn effectively. Previous attempts to fix this used small, static libraries of 3D shapes (like ModelNet40) to create synthetic training data, but these libraries lack the diversity and "messiness" of real-world indoor environments, leading to poor performance when the model is tested on real data.

Methodology

The Prompt3D framework introduces a novel approach to bridge the gap between weak supervision and high-performance detection through two primary modules:

  1. Synthetic Scene Generation (SSG):
    • Prompt-to-Mesh: Instead of relying on a fixed dataset, the authors use ChatGPT to generate diverse text descriptions for 22 object categories (e.g., "an antique wooden door," "a high-back chair"). These prompts are fed into Shap-E, a generative model that converts text into 3D meshes, creating a virtually infinite, diverse library of 3D shapes.
  2. 3D Shape Assembly: The system takes sparse "position-level" labels from a real scene and replaces those points with matching 3D shapes from the generated library. It then applies physics constraints (gravity and collision detection) to ensure the virtual scene looks realistic, resulting in a synthetic scene with "perfect" automatically generated 3D bounding boxes for training.
  3. Prototypical Proposal Feature Alignment (PPFA): To address the "domain gap" between synthetic meshes and real-world sensor data, PPFA focuses on object proposals. It calculates a "prototype" (an average feature vector) for each object class in both synthetic and real domains. By minimizing the distance between these prototypes during training, the model learns features that are robust enough to work on real data despite being trained on virtual shapes.
The Prompt3D pipeline: from random text prompts to synthetic scene generation and final 3D object detection.

Experimental Results

The authors evaluated Prompt3D on the ScanNet benchmark using two popular 3D detectors, VoteNet and GroupFree3D.

  • Significant Performance Gains: Prompt3D improved mean Average Precision ([email protected]) by 5.4% for VoteNet and 8.7% for GroupFree3D compared to the previous state-of-the-art weakly supervised method (BR).
  • Closing the Supervision Gap: With GroupFree3D, the performance gap between this weakly-supervised method (51.2% mAP) and the fully-supervised version (54.2% mAP) was narrowed to just 3%.
  • Qualitative Success: Visualizations demonstrate that Prompt3D successfully localizes objects in complex, cluttered scenes where previous weakly-supervised methods failed or produced inaccurate boxes.
Qualitative comparison of 3D object detection results on ScanNet, comparing baseline methods (WSB, BR) against Prompt3D and Ground Truth.

Conclusion

Prompt3D demonstrates that text-to-3D generative models have matured enough to solve data scarcity in 3D vision. By using generative AI as a "data engine" and aligning features at the object-proposal level, the framework achieves performance near that of fully supervised models with a fraction of the annotation cost. Future work will focus on improving shape fidelity for complex structures and expanding to open-vocabulary detection.

Summary updated

Abstract

The prohibitive cost of annotations for fully supervised 3D indoor object detection limits its practicality. In this work, we propose Random Prompt Assisted Weakly-supervised 3D Object Detection, termed as Prompt3D, a weakly-supervised approach that leverages position-level labels to overcome this challenge. Explicitly, our method focuses on enhancing labeling using synthetic scenes crafted from 3D shapes generated via random prompts. First, a Synthetic Scene Generation (SSG) module is introduced to assemble synthetic scenes with a curated collection of 3D shapes, created via random prompts for each category. These scenes are enriched with automatically generated point-level annotations, providing a robust supervisory frame-work for training the detection algorithm. To enhance the transfer of knowledge from virtual to real datasets, we then introduce a Prototypical Proposal Feature Alignment (PPFA) module. This module effectively alleviates the domain gap by directly minimizing the distance between feature prototypes of the same class proposals across two domains. Compared with sota BR, our method improves by 5.4% and 8.7% on mAP with VoteNet and GroupFree3D serving as detectors respectively, demonstrating the effectiveness of our proposed method. Code is available at: https://github.com/huishengye/prompt3d.
BibTeX
@inproceedings{zhang2024prompt,
  author = {Xiaohong Zhang and Huisheng Ye and Jingwen Li and Qinyu Tang and Yuanqi Li and Yanwen Guo and Jie Guo},
  title = {{Prompt3D: Random Prompt Assisted Weakly-Supervised 3D Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=8nMZ9Qdenn},
}
Figure