ObjectFusion: Multi-modal 3D Object Detection with Object-Centric Fusion

Qi Cai, Yingwei Pan, Ting Yao, Chong-Wah Ngo, Tao Mei

Introduction

In autonomous driving, the integration of LiDAR and camera data is essential for robust 3D object detection. LiDAR provides precise geometric information, while cameras offer rich semantic textures. However, existing multi-modal systems often face a "depth estimation bottleneck." Most current methods transform camera images into a shared Bird’s-Eye-View (BEV) space, a process that relies on error-prone depth prediction for every pixel. This leads to spatial misalignment and "projection distortion," which degrades the fine-grained semantic information from the camera.

Object-Centric Fusion Methodology

ObjectFusion addresses these challenges by shifting from global feature-map warping to an object-centric fusion strategy. Instead of transforming the entire image, the framework follows a "detect then fuse" approach.

Detailed architecture of the ObjectFusion framework, illustrating the multi-modal feature extraction and fusion pipeline.

The process begins with modality-specific encoding, where LiDAR data is processed into voxels and BEV maps, and camera data into 2D image feature maps. A heatmap-based generator identifies 3D object proposals from the BEV features. The core innovation lies in the Object-Centric Projection: 3D proposal boxes are projected back into the 3D voxel grid, the 2D BEV map, and the original 2D image planes. Features are then extracted using Voxel Pooling and RoIAlign, ensuring perfect alignment across sensors for each specific object.

A two-stage fusion process follows. First, a Modality-Specific Context Encoder (MSCE) uses Transformers to capture global context within each modality. Second, the refined features from all sources are concatenated for final classification and box refinement.

Key Insights

A major advantage of ObjectFusion is bypassing depth estimation. By projecting 3D boxes back to 2D images—a mathematically certain operation—the model avoids the primary source of error in multi-modal fusion. Furthermore, extracting features directly from 2D perspective maps preserves the integrity of rich semantic details. To handle calibration misalignments, the authors employ an "RoI Enlarging" trick, doubling the size of the 2D RoI on the image plane to ensure the object is captured even with slight projection errors.

Experimental Results

The model was evaluated on the nuScenes and Waymo Open datasets. ObjectFusion achieved state-of-the-art results, particularly on the nuScenes benchmark.

Quantitative results on the nuScenes dataset, showing ObjectFusion's state-of-the-art performance.

On the nuScenes validation set, ObjectFusion reached 69.8% mAP and 72.3% NDS, outperforming BEVFusion by 1.3% mAP. On the test set, it established a new state-of-the-art with 71.0% mAP. Notably, the model showed significant gains in challenging conditions, such as a 3.2% improvement in night scenarios over BEVFusion, demonstrating the reliability of object-centric alignment in noisy environments.

Conclusion and Future Work

While ObjectFusion sets a new performance bar, it introduces a computational trade-off. The inference time is approximately 274ms per sample on an NVIDIA V100, slightly slower than BEVFusion's 257ms due to sequential feature extraction. Future work will focus on parallelizing this process for real-time deployment. Additionally, as the system relies on initial BEV-based proposals, improving the recall of the first stage remains a key area for development.

Summary updated

Abstract

Recent progress on multi-modal 3D object detection has featured BEV (Bird-Eye-View) based fusion, which effectively unifies both LiDAR point clouds and camera images in a shared BEV space. Nevertheless, it is not trivial to perform camera-to-BEV transformation due to the inherently ambiguous depth estimation of each pixel, resulting in spatial misalignment between these two multi-modal features. Moreover, such transformation also inevitably leads to projection distortion of camera image features in BEV space. In this paper, we propose a novel Object-centric Fusion (ObjectFusion) paradigm, which completely gets rid of camera-to-BEV transformation during fusion to align object-centric features across different modalities for 3D object detection. ObjectFusion first learns three kinds of modality-specific feature maps (i.e., voxel, BEV, and image features) from LiDAR point clouds and its BEV projections, camera images. Then a set of 3D object proposals are produced from the BEV features via a heatmap-based proposal generator. Next, the 3D object proposals are reprojected back to voxel, BEV, and image spaces. We leverage voxel and RoI pooling to generate spatially aligned object-centric features for each modality. All the object-centric features of three modalities are further fused at object level, which is finally fed into the detection heads. Extensive experiments on nuScenes dataset demonstrate the superiority of our ObjectFusion, by achieving 69.8% mAP on nuScenes validation set and improving BEVFusion by 1.3%.
BibTeX
@inproceedings{cai2023objectfusion,
  author = {Qi Cai and Yingwei Pan and Ting Yao and Chong-Wah Ngo and Tao Mei},
  title = {{ObjectFusion: Multi-modal 3D Object Detection with Object-Centric Fusion}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01656},
}
Figure