FGFusion: Fine-Grained Lidar-Camera Fusion for 3D Object Detection

Zixuan Yin, Han Sun, Ningzhong Liu, Huiyu Zhou, Jiaquan Shen

Abstract

FGFusion is a novel multi-modal framework designed to enhance 3D object detection by fusing Lidar and camera data across multiple scales. Traditional fusion methods often suffer from a "coarse-grained" problem, where progressive downsampling for high-level semantics leads to the loss of critical low-level spatial details. FGFusion addresses this by preserving fine-grained information through a dual-pathway image hierarchy and an auxiliary Lidar network. Experimental results on the KITTI and Waymo Open Datasets demonstrate that this approach achieves state-of-the-art performance, particularly for small or occluded targets.

Introduction

In the context of autonomous driving, 3D object detection relies on the synergy between Lidar and cameras. While Lidar provides precise depth information, it lacks resolution; conversely, cameras provide rich texture and shape but lack inherent depth.

Most existing fusion architectures progressively downsample data to extract high-level semantic features (the "what") before fusion. However, this process inevitably discards low-level detailed information (the "where" and "fine structure"). This loss can lead to poor performance on small or occluded targets, sometimes causing fusion models to underperform compared to Lidar-only baselines. FGFusion aims to bridge this gap by ensuring fine-grained details are preserved and utilized throughout the fusion process.

Methodology

The FGFusion architecture introduces three key innovations to maintain and integrate fine-grained information:

  1. Dual Pathway Image Hierarchy: Instead of a simple feature extractor, the image branch utilizes a top-down pathway for semantics and a bottom-up attention pathway. This "Attention Pyramid" combines spatial attention to locate identifiable regions and channel attention to pass detailed information to higher levels.
  2. Auxiliary Lidar Network: To prevent the 3D backbone from losing spatial structure during downsampling, an auxiliary network is used during training. This network provides point-level supervision via foreground segmentation and center estimation. Crucially, this network is discarded during inference, improving accuracy without adding computational overhead for real-time operation.
  3. Multi-Scale Fusion (MSF): Rather than fusing only the final, most downsampled feature map, FGFusion selects the last $N$ feature maps from both the image and Lidar pyramids. It employs a cross-attention mechanism to fuse these multi-scale features, leveraging both abstract semantics and granular details.
The FGFusion architecture, highlighting the Dual Pathway image branch, the Attention Pyramid, and the Auxiliary Lidar Network.

Experimental Results

The model was evaluated on two industry-standard benchmarks: the KITTI dataset and the Waymo Open Dataset. Performance was measured across Car, Pedestrian, and Cyclist categories at various difficulty levels.

KITTI Performance

FGFusion achieved a mean AP (mAP) of 77.05, the highest among compared state-of-the-art methods. It demonstrated particular dominance in the "Hard" difficulty category, proving its robustness in complex scenes. As shown in the results table, FGFusion consistently outperforms both Lidar-only and previous fusion-based methods across nearly all categories.

Comparison of FGFusion against state-of-the-art Lidar-only and fusion methods on the KITTI benchmark.

Waymo Open Dataset

The model outperformed prominent fusion methods like DeepFusion and PointAugmenting. In the challenging LEVEL2 difficulty for vehicles, it achieved 73.34 APH. Notably, the fine-grained approach proved exceptionally effective for small objects; on Waymo, FGFusion improved pedestrian recognition (APH) by 18.53% compared to the Lidar-only PV-RCNN baseline.

Discussion and Key Insights

Ablation studies confirmed that the Multi-Scale Fusion (MSF) component is the core driver of success, as its removal resulted in a performance drop of approximately 1.7-1.9% APH. The authors also identified a "sweet spot" for fusion complexity: using two or three layers for fusion provides a significant boost, but benefits plateau at four layers, suggesting a limit to the utility of redundant low-level information.

While the auxiliary network is removed at inference, the multi-scale cross-attention mechanism does add some complexity compared to single-layer fusion. Furthermore, like most fusion methods, FGFusion remains sensitive to sensor calibration. Future work may involve optimizing attention blocks for edge-device deployment and increasing robustness to calibration drift.

Conclusion

FGFusion demonstrates that preserving fine-grained spatial and geometric details is essential for high-performance 3D object detection. By integrating multi-scale features and using auxiliary training signals, the framework sets a new standard for Lidar-camera fusion, particularly in the detection of small and challenging objects in autonomous driving environments.

Summary updated

BibTeX
@inproceedings{yin2023fgfusion,
  author = {Zixuan Yin and Han Sun and Ningzhong Liu and Huiyu Zhou and Jiaquan Shen},
  title = {{FGFusion: Fine-Grained Lidar-Camera Fusion for 3D Object Detection}},
  booktitle = {PRCV},
  year = {2023},
  doi = {10.1007/978-981-99-8435-0\_40},
}

Loading related content...

Figure