SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detection

Yichen Xie, Chenfeng Xu, Marie-Julie Rakotosaona, Patrick Rim, Federico Tombari, Kurt Keutzer, Masayoshi Tomizuka, Wei Zhan

Introduction

In the field of autonomous driving, perceiving the 3D environment with high precision and low latency is a fundamental challenge. Modern systems typically rely on a combination of LiDAR for precise geometry and cameras for rich semantic detail. However, fusing these disparate data types has traditionally been computationally expensive. SparseFusion, presented by researchers from UC Berkeley, Google, and Caltech, introduces a paradigm shift as the first multi-sensor 3D object detection method to exclusively use sparse candidates and representations for fusion, achieving state-of-the-art accuracy with industry-leading inference speeds.

Problem Statement: The Inefficiency of Dense Fusion

Most existing LiDAR-camera fusion methods rely on dense representations, such as projecting image features into a 3D Bird's-Eye View (BEV) grid or augmenting every point in a LiDAR cloud with image data. This approach is inherently inefficient because objects typically occupy only a tiny fraction of a 3D scene. Generating dense representations for empty space is computationally wasteful and introduces background noise that can degrade detection performance. Furthermore, aligning these dense modalities is often the primary bottleneck for real-time performance.

Methodology: Sparse-to-Sparse Fusion

SparseFusion introduces a "sparse-to-sparse" pipeline that operates on instance-level features rather than global scene grids.

The SparseFusion framework utilizes parallel branches to extract sparse candidates for fusion in a unified 3D space.

Parallel Sparse Detection

Instead of early-stage raw data fusion, SparseFusion runs parallel, modality-specific detectors. The LiDAR branch uses a voxel-based backbone, while the camera branch employs a deformable-DETR style architecture. Each branch produces a set of sparse instance features (candidates) representing potential objects.

Cross-Modality Information Transfer

To mitigate the inherent weaknesses of each sensor, the authors introduce two novel modules:

  • Geometric Transfer (LiDAR to Camera): Projects LiDAR points onto images to create sparse depth maps, which are used to generate depth-aware image features for better 3D localization.
  • Semantic Transfer (Camera to LiDAR): Projects LiDAR points into image feature space to "paint" LiDAR BEV features with semantic information, helping the LiDAR branch distinguish between geometrically similar objects.

Disentangled Transformation and Fusion

Camera-based candidates are transformed into the LiDAR coordinate space by disentangling their representations—recognizing that while an object's class is view-invariant, its 3D bounding box is view-dependent. Finally, a lightweight self-attention module fuses these sparse candidates from both modalities in a unified 3D space.

Experimental Results

The model was trained and evaluated on the nuScenes dataset, a large-scale benchmark featuring 1,000 scenes with 32-beam LiDAR and six multi-view cameras.

Accuracy and Efficiency

SparseFusion sets a new benchmark for the performance-speed trade-off. On the nuScenes test set, it achieved 73.8% NDS and 72.0% mAP, outperforming previous leaders such as DeepInteraction and BEVFusion.

SparseFusion achieves state-of-the-art mAP while maintaining significantly higher FPS compared to existing dense fusion methods.

In terms of efficiency, SparseFusion runs at 5.6 FPS on an NVIDIA A6000 GPU, which is approximately 1.8x faster than DeepInteraction (3.1 FPS) and significantly faster than dense fusion methods. Qualitative results show that the model is particularly effective at detecting small or distant objects where LiDAR data is sparse, thanks to the semantic richness of the camera branch.

Conclusion

SparseFusion demonstrates that focusing on object candidates rather than dense scene grids significantly improves both efficiency and accuracy in 3D object detection. By utilizing parallel sparse detection and cross-modality information transfer, the model achieves state-of-the-art results on the nuScenes benchmark while maintaining real-time performance. Future work may involve extending this sparse-to-sparse paradigm to temporal fusion and other dense perception tasks.

Summary updated

Abstract

By identifying four important components of existing LiDAR-camera 3D object detection methods (LiDAR and camera candidates, transformation, and fusion outputs), we observe that all existing methods either find dense candidates or yield dense representations of scenes. However, given that objects occupy only a small part of a scene, finding dense candidates and generating dense representations is noisy and inefficient. We propose SparseFusion, a novel multi-sensor 3D detection method that exclusively uses sparse candidates and sparse representations. Specifically, SparseFusion utilizes the outputs of parallel detectors in the LiDAR and camera modalities as sparse candidates for fusion. We transform the camera candidates into the LiDAR coordinate space by disentangling the object representations. Then, we can fuse the multi-modality candidates in a unified 3D space by a lightweight self-attention module. To mitigate negative transfer between modalities, we propose novel semantic and geometric cross-modality transfer modules that are applied prior to the modality-specific detectors. SparseFusion achieves state-of-the-art performance on the nuScenes benchmark while also running at the fastest speed, even outperforming methods with stronger backbones. We perform extensive experiments to demonstrate the effectiveness and efficiency of our modules and overall method pipeline. Our code will be made publicly available at https://github.com/yichen928/SparseFusion.
BibTeX
@inproceedings{xie2023sparsefusion,
  author = {Yichen Xie and Chenfeng Xu and Marie-Julie Rakotosaona and Patrick Rim and Federico Tombari and Kurt Keutzer and Masayoshi Tomizuka and Wei Zhan},
  title = {{SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure