Introduction
In autonomous driving, the integration of LiDAR and camera data is essential for robust 3D object detection. LiDAR provides precise geometric information, while cameras offer rich semantic textures. However, existing multi-modal systems often face a "depth estimation bottleneck." Most current methods transform camera images into a shared Bird’s-Eye-View (BEV) space, a process that relies on error-prone depth prediction for every pixel. This leads to spatial misalignment and "projection distortion," which degrades the fine-grained semantic information from the camera.
Object-Centric Fusion Methodology
ObjectFusion addresses these challenges by shifting from global feature-map warping to an object-centric fusion strategy. Instead of transforming the entire image, the framework follows a "detect then fuse" approach.
The process begins with modality-specific encoding, where LiDAR data is processed into voxels and BEV maps, and camera data into 2D image feature maps. A heatmap-based generator identifies 3D object proposals from the BEV features. The core innovation lies in the Object-Centric Projection: 3D proposal boxes are projected back into the 3D voxel grid, the 2D BEV map, and the original 2D image planes. Features are then extracted using Voxel Pooling and RoIAlign, ensuring perfect alignment across sensors for each specific object.
A two-stage fusion process follows. First, a Modality-Specific Context Encoder (MSCE) uses Transformers to capture global context within each modality. Second, the refined features from all sources are concatenated for final classification and box refinement.
Key Insights
A major advantage of ObjectFusion is bypassing depth estimation. By projecting 3D boxes back to 2D images—a mathematically certain operation—the model avoids the primary source of error in multi-modal fusion. Furthermore, extracting features directly from 2D perspective maps preserves the integrity of rich semantic details. To handle calibration misalignments, the authors employ an "RoI Enlarging" trick, doubling the size of the 2D RoI on the image plane to ensure the object is captured even with slight projection errors.
Experimental Results
The model was evaluated on the nuScenes and Waymo Open datasets. ObjectFusion achieved state-of-the-art results, particularly on the nuScenes benchmark.
On the nuScenes validation set, ObjectFusion reached 69.8% mAP and 72.3% NDS, outperforming BEVFusion by 1.3% mAP. On the test set, it established a new state-of-the-art with 71.0% mAP. Notably, the model showed significant gains in challenging conditions, such as a 3.2% improvement in night scenarios over BEVFusion, demonstrating the reliability of object-centric alignment in noisy environments.
Conclusion and Future Work
While ObjectFusion sets a new performance bar, it introduces a computational trade-off. The inference time is approximately 274ms per sample on an NVIDIA V100, slightly slower than BEVFusion's 257ms due to sequential feature extraction. Future work will focus on parallelizing this process for real-time deployment. Additionally, as the system relies on initial BEV-based proposals, improving the recall of the first stage remains a key area for development.
Summary updated