FlatFusion: Delving into Details of Sparse Transformer-based Camera-LiDAR Fusion for Autonomous Driving

Yutao Zhu, Xiaosong Jia, Xinyu Yang, Junchi Yan

Introduction

Autonomous driving systems rely heavily on the effective fusion of camera and LiDAR data. While cameras provide rich semantic information, they lack precise depth. Conversely, LiDAR offers accurate geometric data but is sparse and semantically limited. Recent advancements have favored Sparse Transformers for their efficiency over traditional dense grids. However, many existing methods overlook critical architectural "micro-details," such as data mapping and token grouping.

FlatFusion addresses this gap by systematically optimizing these design principles. The framework achieves a state-of-the-art 73.7 NDS on the nuScenes benchmark while maintaining a real-time inference speed of 10.1 FPS.

Methodology

FlatFusion is built upon several optimized design principles aimed at maximizing both accuracy and computational efficiency. The architecture systematically handles the transition between 2D image space and 3D LiDAR space.

FlatFusion Architecture Overview

Optimized Projection and Fusion

  • Voxel-based 3D-to-2D Projection: Unlike "pillar-based" methods that collapse the vertical Z-axis, FlatFusion utilizes voxels. This preserves height information, ensuring LiDAR features are comprehensively aligned with image data.
  • Partial Projection for 2D-to-3D: To avoid the latency of full-depth estimation, FlatFusion projects only image patches corresponding to actual LiDAR points. This "Partial Projection" is faster and eliminates "bleeding" artifacts common in nearest-neighbor interpolation.
  • 3D-to-2D First Order: The authors found that fusing 3D features into the 2D image space before the reverse process preserves more fine-grained detail, which is typically lost during network downsampling.

Efficient Attention Mechanisms

  • Flatten Window Partition: To manage sparse tokens, the "Flatten Window" algorithm groups tokens by their indices rather than strict spatial coordinates. This approach is significantly faster on modern GPUs while maintaining high accuracy.
  • Micro-structure Refinements: The model employs a PreNorm structure with 3D Position Encoding (PE) and an optimized group size of 80 tokens to balance the receptive field with local detail.

Experimental Setup

The framework was evaluated using the nuScenes dataset, which includes 1,000 scenes equipped with 6 cameras and 1 LiDAR sensor. The dataset contains 1.4 million annotated 3D bounding boxes across 23 object classes, providing a robust benchmark for autonomous driving tasks.

Results and Discussion

FlatFusion demonstrates a superior trade-off between performance and speed compared to existing state-of-the-art models.

Performance (NDS) vs. Inference Speed (FPS) on nuScenes

Key Performance Metrics

  • Accuracy: FlatFusion reached 73.7 NDS and 70.6 mAP on the validation set, and 74.7 NDS on the test set.
  • Efficiency: Running at 10.1 FPS on an NVIDIA RTX 4090, it outperforms models like UniTR (9.3 FPS) and CMT (6.4 FPS). Notably, it surpasses UniTR by 0.4 NDS despite using a simpler ResNet-18 backbone.

Insights

The study highlights that maintaining the vertical dimension ("Z-axis") through voxel projection is a critical performance driver. Furthermore, the sequence of projections and the use of index-based token grouping (Flatten Window) prove that architectural simplicity and hardware-aware design can yield significant gains in sparse multi-modal fusion.

Conclusion and Future Work

While FlatFusion sets a new standard for sparse fusion, the "partition time" for grouping tokens remains a primary latency bottleneck. Future research will focus on optimizing the BEV backbone and detection heads to further increase frame rates, potentially enabling deployment on lower-power edge devices for production vehicles.

Summary updated

BibTeX
@inproceedings{zhu2025flatfusion,
  author = {Yutao Zhu and Xiaosong Jia and Xinyu Yang and Junchi Yan},
  title = {{FlatFusion: Delving into Details of Sparse Transformer-based Camera-LiDAR Fusion for Autonomous Driving}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2025},
  doi = {10.1109/icra55743.2025.11128084},
}
Figure