HGSFusion: Radar-Camera Fusion with Hybrid Generation and Synchronization for 3D Object Detection

Zijian Gu, Jianwei Ma, Yan Huang, Honghao Wei, Zhanye Chen, Hui Zhang, Wei HongAAAI 2025 · DOI

Introduction

3D object detection is a cornerstone of autonomous driving. While LiDAR provides high-precision dense point clouds, its high cost and vulnerability to adverse weather conditions like fog or heavy rain limit its ubiquity. Millimeter-wave (mmWave) radar offers a robust, all-weather alternative at a lower cost. However, radar data is inherently sparse and suffers from Direction-of-Arrival (DOA) estimation errors, where detected points deviate from their true spatial locations due to signal processing limitations.

Radar point sparsity and DOA estimation errors on a vehicle image.

HGSFusion addresses these challenges by integrating camera and radar data through a novel hybrid generation and dual synchronization framework. By leveraging the semantic richness of images to guide radar densification and synchronizing features across modalities, HGSFusion achieves superior detection performance, particularly in low-visibility environments.

Methodology

The HGSFusion architecture consists of two core components designed to mitigate radar's limitations: the Radar Hybrid Generation Module (RHGM) and the Dual Synchronization Module (DSM).

Overview of the HGSFusion architecture, featuring the RHGM and DSM modules.

Radar Hybrid Generation Module (RHGM)

The RHGM densifies sparse radar data using image-derived semantic information. First, raw radar points are projected onto the image plane. Using semantic segmentation masks, the system identifies "foreground" points belonging to objects. To account for DOA errors and sparsity, the module generates synthetic points:

  • Gaussian Distribution: New points are sampled near existing detections to model local angular uncertainty.
  • Uniform Distribution: Points are distributed across the object mask to ensure spatial coverage.

These points are encoded separately (raw, foreground, and generated) to allow the network to learn their relative reliability and weigh them differently during feature extraction.

Dual Synchronization Module (DSM)

The DSM ensures effective multi-modal feature fusion through two stages:

  1. Spatial Synchronization: Radar's high positional confidence guides image features, focusing the model's attention on regions likely to contain objects. It employs atrous (dilated) convolutions to expand the receptive field, ensuring large objects are captured accurately.
  2. Modality Synchronization: A dynamic weighting mechanism evaluates the importance of radar versus image features. In scenarios where image quality is poor (e.g., nighttime or glare), the module learns to prioritize radar data, preventing "negative fusion" where noisy visual data might otherwise degrade performance.

Experimental Results

HGSFusion was evaluated on two prominent 4D millimeter-wave radar benchmarks: View of Delft (VoD) and TJ4DRadSet.

Performance on VoD

On the VoD dataset, HGSFusion achieved a mean Average Precision (mAP) of 58.96% in the Entire Annotated Area, a 2.65% improvement over the previous state-of-the-art, LXL. In the Region of Interest (RoI) or "Driving Corridor," the improvement was even more pronounced at 6.53%. The model showed particular strength in vehicle detection, yielding an 8.79% improvement over competitors like TL-4DRCF.

Robustness to Lighting Conditions

Testing on TJ4DRadSet demonstrated HGSFusion's resilience across various lighting conditions. In "Dark" scenes, the model improved 3D mAP by 11.41% compared to a standard fusion baseline. This highlights the effectiveness of the Modality Sync component in maintaining perception capabilities when visual sensors fail.

Conclusion

HGSFusion effectively bridges the gap between sparse, noisy radar data and high-resolution imagery. By intelligently generating hybrid points and synchronizing multi-modal features, it sets a new standard for 4D radar-camera fusion. Future work will focus on refining point generation for smaller, complex geometries like cyclists and pedestrians to further enhance detection accuracy across all object categories.

Summary updated

BibTeX
@inproceedings{gu2025hgsfusion,
  author = {Zijian Gu and Jianwei Ma and Yan Huang and Honghao Wei and Zhanye Chen and Hui Zhang and Wei Hong},
  title = {{HGSFusion: Radar-Camera Fusion with Hybrid Generation and Synchronization for 3D Object Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i3.32328},
}
Figure