SuperFusion: Multilevel LiDAR-Camera Fusion for Long-Range HD Map Generation

Hao Dong, Weihao Gu, Xianjing Zhang, Jintao Xu, Rui Ai, Huimin Lu, Juho Kannala, Xieyuanli Chen

TL;DR

SuperFusion is a novel multilevel LiDAR-camera fusion framework that extends online high-definition (HD) map generation from the standard 30-meter range to a much more practical 90 meters. By integrating sensor data at the data, feature, and bird's-eye-view (BEV) levels, it provides the long-range environmental context necessary for smoother, safer autonomous path planning.

What Problem Does This Paper Address?

High-definition (HD) maps—containing semantic layers like lane boundaries, dividers, and pedestrian crossings—are the lifeblood of autonomous vehicle navigation. While traditional HD maps are built offline through labor-intensive manual annotation, recent research has shifted toward "online" generation using onboard sensors.

However, existing online methods suffer from a major limitation: range. Most current models only predict maps within a 30-meter radius. For a vehicle traveling at highway speeds, 30 meters is covered in about a second, leading to jerky path planning, frequent replanning, and potential safety risks. The challenge lies in the sensors themselves: LiDAR provides precise depth but has a limited effective range for ground-plane measurements (often around 30m for common sensors), while cameras see much further but struggle with accurate depth estimation.

Methods

The authors propose SuperFusion, which tackles the range problem through a three-tiered fusion strategy:

  1. Data-level Fusion: The system projects sparse LiDAR point clouds onto camera images to create a depth prior. This "sparse depth map" is used to supervise a depth-aware camera-to-BEV transformation, significantly improving the accuracy of how image features are projected into 3D space.
  2. Feature-level Fusion (Image-Guided LiDAR Prediction): This is the core innovation for long-range mapping. Since LiDAR data is often missing at distances beyond 30m, SuperFusion uses a cross-attention mechanism. It treats camera features as a guide to "predict" or hallucinate what the LiDAR features would look like at long ranges (up to 90m).
  3. BEV-level Fusion: To account for calibration errors or depth estimation inaccuracies, the model employs a BEV alignment module. It learns a "flow field" to warp and align the camera-derived BEV features with the LiDAR-derived features before final concatenation.

The final fused features are passed through a decoder with three heads: semantic segmentation, instance embedding (to distinguish individual lanes), and direction prediction (to vectorize the map).

Fig. 2: Pipeline overview of SuperFusion. Our method fuses camera and LiDAR data in three levels: the data-level fusion fuses depth information from LiDAR to improve the accuracy of image depth estimation, the feature-level fusion uses cross-attention for long-range LiDAR BEV feature prediction with the guidance of image features, and the BEV-level fusion aligns two branches to generate high-quality fused BEV features. Finally, the fused BEV features can support different heads, including semantic segmentation, instance embedding, and direction prediction, and finally post-processed to generate the HD map prediction.

Datasets

The researchers evaluated SuperFusion on two primary datasets:

  • nuScenes: A large-scale, industry-standard public dataset for autonomous driving.
  • Self-recorded Dataset: A custom dataset containing 21,000 frames of real-world driving data, featuring hand-labeled lane boundaries and dividers to test the model's generalization in diverse environments.

Key Insights and Ideas

  • Complementary Strengths: The "aha moment" is recognizing that while LiDAR is physically limited in range for ground detection, the patterns it detects at short range can be extrapolated to long range using the visual context provided by cameras.
  • Alignment Matters: Direct concatenation of sensor features often fails due to slight misalignments in sensor calibration. The learned "flow field" in the BEV alignment module (shown in Figure 4) is critical for maintaining high performance.
  • Downstream Impact: The paper emphasizes that perception isn't an end in itself. By extending the map to 90m, the vehicle can plan smoother trajectories, avoiding the "non-smooth paths" that occur when a model can only see 30m ahead.
Fig. 1: Long-range HD map generation for path planning. The red car represents the current position of the car, and the blue star is the goal. The upper figure shows that the baseline method only generates short-range HD maps, leading to lousy planning results. The lower one shows that our SuperFusion generates accurate HD maps in both short and long ranges, which serves online path planning well for autonomous driving.

Key Results

SuperFusion significantly outperformed state-of-the-art baselines like HDMapNet and BEVFusion:

  • Superior Accuracy: On the nuScenes dataset, SuperFusion achieved an average IoU (Intersection over Union) for lane dividers of 38.0%, compared to 30.5% for HDMapNet.
  • Long-Range Performance: In the 60–90m interval—where most models fail—SuperFusion maintained a divider IoU of 29.2%, whereas HDMapNet dropped to 18.1% and LiDAR-only methods plummeted to 4.4%.
  • Planning Success: In a downstream path-planning simulation, SuperFusion achieved a 72% success rate, a massive jump from the 45-49% success rates of existing methods.

Limitations and Future Work

While SuperFusion shows remarkable progress, the authors note that depth estimation remains a challenging bottleneck; inaccurate depth can still lead to alignment failures. Future work could involve exploring more sophisticated temporal fusion (using multiple frames over time) to further stabilize long-range predictions or expanding the map to include dynamic objects and more complex intersection geometries.

Summary updated

Abstract

High-definition (HD) semantic map generation of the environment is an essential component of autonomous driving. Existing methods have achieved good performance in this task by fusing different sensor modalities, such as LiDAR and camera. However, current works are based on raw data or network feature-level fusion and only consider short-range HD map generation, limiting their deployment to realistic autonomous driving applications. In this paper, we focus on the task of building the HD maps in both short ranges, i.e., within 30 m, and also predicting long-range HD maps up to 90 m, which is required by downstream path planning and control tasks to improve the smoothness and safety of autonomous driving. To this end, we propose a novel network named SuperFusion, exploiting the fusion of LiDAR and camera data at multiple levels. We use LiDAR depth to improve image depth estimation and use image features to guide long-range LiDAR feature prediction. We benchmark our SuperFusion on the nuScenes dataset and a self-recorded dataset and show that it outperforms the state-of-the-art baseline methods with large margins on all intervals. Additionally, we apply the generated HD map to a downstream path planning task, demonstrating that the long-range HD maps predicted by our method can lead to better path planning for autonomous vehicles. Our code has been released at https://github.com/haomo-ai/SuperFusion.
BibTeX
@inproceedings{dong2024superfusion,
  author = {Hao Dong and Weihao Gu and Xianjing Zhang and Jintao Xu and Rui Ai and Huimin Lu and Juho Kannala and Xieyuanli Chen},
  title = {{SuperFusion: Multilevel LiDAR-Camera Fusion for Long-Range HD Map Generation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10611320},
}
Figure