Abstract
In this paper, we focus on exploring the fusion of images and point clouds for 3D object detection in view of the complementary nature of the two modalities, i.e., images possess more semantic information while point clouds specialize in distance sensing. To this end, we present a novel two-stage multi-modal fusion network for 3D object detection, taking both binocular images and raw point clouds as input. The whole architecture facilitates two-stage fusion. The first stage aims at producing 3D proposals through sparse point-wise feature fusion. Within the first stage, we further exploit a joint anchor mechanism that enables the network to utilize 2D-3D classification and regression simultaneously for better proposal generation. The second stage works on the 2D and 3D proposal regions and fuses their dense features. In addition, we propose to use pseudo LiDAR points from stereo matching as a data augmentation method to densify the LiDAR points, as we observe that objects missed by the detection network mostly have too few points especially for far-away objects. Our experiments on the KITTI dataset show that the proposed multi-stage fusion helps the network to learn better representations.
Introduction
3D object detection is a cornerstone of autonomous driving, yet it remains a difficult balancing act. LiDAR sensors provide precise geometric data but produce sparse, unordered point clouds that struggle to capture fine-grained semantic details. Conversely, 2D images are rich in color and texture but lack reliable depth information.
Existing fusion methods often fall short: some project LiDAR data into 2D views, losing critical 3D information, while others only combine modalities at the extreme ends of the pipeline. Furthermore, distant objects often reflect only a handful of LiDAR points, making them nearly invisible to standard 3D detectors. This research aims to bridge this gap by creating a deeply fused architecture that leverages the strengths of both sensors at multiple stages.
Methodology
The proposed framework utilizes a two-stage multi-modal fusion network designed to integrate binocular images and raw point clouds effectively.
Stage 1: Proposal Generation
In the first stage, the network generates 3D proposals using sparse point-wise feature fusion. It employs PointNet++ to process raw LiDAR points and a ResNet-50/FPN backbone for images. Each 3D point is projected into the 2D image feature map to extract semantic features, creating a hybrid representation. A key innovation here is the Joint Anchor Mechanism, where 3D anchors are seeded and projected onto the image to create 2D counterparts, allowing the network to utilize 2D-3D classification and regression simultaneously.
Stage 2: Box Refinement
The second stage focuses on RoI-wise fusion. The network extracts dense features from both the 3D point regions and the 2D image Regions of Interest (RoI). These features are concatenated and processed through a light-weight PointNet to refine final box coordinates and confidence scores.
Reprojection Loss and Pseudo-LiDAR
A novel Reprojection Loss is introduced to force 3D bounding box proposals to align with 2D image constraints. Since 2D image boundaries are often sharper than sparse 3D point clouds, this coupling ensures geometric consistency across modalities.
To address the sparsity of distant objects, the authors employ Pseudo-LiDAR Augmentation. By using stereo matching from binocular cameras, they generate "pseudo-LiDAR" points to densify the scene, providing the network with more data to detect far-away vehicles that standard LiDAR might miss.
Experimental Results
The model was evaluated on the KITTI 2D/3D Object Detection benchmark, focusing on the "Car" category across Easy, Moderate, and Hard difficulty levels.
- Detection Performance: The method achieved 87.22% Average Precision (AP) on the "Easy" category for 3D detection, outperforming established models like PointRCNN (86.96%) and F-PointNet (81.20%).
- Ablation Study: The Reprojection Loss alone provided a significant +4.07% AP boost on the Easy 3D detection task, validating the effectiveness of the 2D-3D coupling.
- Robustness: The use of pseudo-LiDAR points significantly improved performance in "Moderate" and "Hard" cases, where objects are typically further away or partially occluded.
Conclusion
This research demonstrates that multi-stage fusion—at both the point level and the RoI level—enables a network to learn superior representations by combining local semantic cues with global geometric consistency. While pseudo-LiDAR augmentation effectively "patches" sparse data for distant objects, future work will focus on mitigating noise from stereo matching and optimizing the pipeline for real-time performance on embedded hardware.
Summary updated