Long-Tailed 3D Detection via 2D Late Fusion

Yechi Ma, Neehar Peri, Shuoquan Wei, Yanan Li, Wei Hua, Deva Ramanan, Shu KongOpenReview 2024 · Submission to ICLR 2024 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Autonomous vehicles (AVs) must accurately detect objects from both common and rare classes for safe navigation, motivating the problem of Long-Tailed 3D Object Detection (LT3D). Contemporary LiDAR-based 3D detectors perform poorly on rare classes (e.g., CenterPoint achieves only 5.1 AP on stroller) because it is difficult to recognize objects from sparse LiDAR points alone. RGB images may help resolve such ambiguities, motivating the study of multi-modal RGB-LiDAR fusion. Specifically, we delve into a simple late-fusion framework that ensembles 2D RGB and 3D LiDAR detections and find that (a) high-resolution RGB images help recognize rare objects, (b) LiDAR provides precise 3D localization, and (c) uni-modal detectors can easily leverage more training data because they do not require aligning and annotating multi-modal data. We examine three critical components in this late-fusion framekwork: (1) whether to train 2D or 3D RGB detectors, (2) whether to match RGB and LiDAR detections in 3D or the projected 2D image plane (3) how to fuse matched detections. Extensive experiments reveal that using 2D RGB detectors, matching in the 2D image plane, and fusing scores probabilistically with calibration leads to the state-of-the-art LT3D performance, achieving 51.4 mAP on the established nuScenes LT3D benchmark, improving over prior work by 5.9 mAP.
BibTeX
@misc{ma2024long,
  author = {Yechi Ma and Neehar Peri and Shuoquan Wei and Yanan Li and Wei Hua and Deva Ramanan and Shu Kong},
  title = {{Long-Tailed 3D Detection via 2D Late Fusion}},
  year = {2024},
  url = {https://openreview.net/forum?id=XM0fepbZE9},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}

Loading related content...

Figure