Towards Dense and Accurate Radar Perception Via Efficient Cross-Modal Diffusion Model

Ruibin Zhang, Donglai Xue, Yuhan Wang, Ruixu Geng, Fei Gao

Abstract

Millimeter wave (mmWave) radars are robust in extreme weather but suffer from sparsity and noise, limiting their use in micro aerial vehicle (MAV) navigation. This paper presents a cross-modal diffusion model that predicts dense, LiDAR-like point clouds from raw radar data. By integrating consistency model distillation, we achieve an 80x inference speedup, enabling real-time performance on resource-constrained MAVs. Extensive evaluations on public and custom datasets demonstrate superior accuracy and generalization.

Introduction

Millimeter-wave (mmWave) radars are essential for "all-weather" robotics, maintaining visibility in fog, smoke, and dust where LiDAR and cameras fail. However, traditional signal processing (e.g., CFAR) yields sparse, noisy point clouds unsuitable for complex obstacle avoidance or SLAM. While deep learning has attempted to upsample radar data, existing GAN or U-Net approaches often lack structural fidelity or fail in unseen environments. Furthermore, high-fidelity generative models like Diffusion Models are typically too computationally intensive for real-time MAV navigation.

Methodology

We frame radar-to-LiDAR translation as an image restoration task, drawing inspiration from low-resolution and noisy image denoising.

Data Representation and Framework

Raw radar signals are converted into Range-Azimuth Heatmaps (RAH), while LiDAR ground truth is projected into Bird’s Eye View (BEV) images. We employ a diffusion model based on the EDM framework to map the impaired radar domain to the dense LiDAR domain. During training, the model learns to "hallucinate" the density of LiDAR by using it as a ground-truth supervisor.

Overview of the proposed approach, framing radar-to-LiDAR translation as a denoising problem and utilizing consistency sampling for acceleration.

Loss Functions and Acceleration

To prioritize structural integrity (e.g., straight walls), we combine Mean Squared Error (MSE) with Learned Perceptual Image Patch Similarity (LPIPS). To overcome the latency of iterative diffusion, we distill the model into a Consistency Model. This allows the network to generate a final prediction from noise in a single step, drastically reducing inference time without significant loss in visual quality.

Experimental Evaluation

We validated our approach using the ColoRadar dataset (52 sequences across mines, labs, and outdoors) and a self-made dataset collected with a custom handheld rig featuring an Nvidia Jetson Orin NX, Livox Mid-360 LiDAR, and a TI-AWR1843BOOST radar.

Performance and Generalization

Our method consistently outperformed baselines like OS-CFAR, RadarHD, and RPDNet in Chamfer Distance (CD) and Hausdorff Distance (HD). Notably, the model generalized well to entirely unseen sensors and environments, maintaining high structural accuracy where traditional methods failed.

Qualitative comparison showing the proposed method's ability to reconstruct dense environments from radar data, even in unseen settings, compared to traditional methods.

Real-Time Deployment

On the Nvidia Jetson Orin NX, the consistency model achieved an inference time of 181.6 ms (~5 Hz). This represents an 80x speedup over the standard iterative diffusion process, making it viable for real-time MAV navigation.

Conclusion and Future Work

This work demonstrates that cross-modal diffusion, combined with consistency distillation, can bridge the gap between low-cost radar and high-fidelity LiDAR perception. Future research will focus on extending the model to 3D point cloud generation, handling dynamic obstacles, and improving performance on low-reflectivity materials like vegetation.

Summary updated

Abstract

Millimeter wave (mmWave) radars have attracted significant attention from both academia and industry due to their capability to operate in extreme weather conditions. However, they face challenges in terms of sparsity and noise interference, which hinder their application in the field of micro aerial vehicle (MAV) autonomous navigation. To this end, this paper proposes a novel approach to dense and accurate mmWave radar point cloud construction via cross-modal learning. Specifically, we introduce diffusion models, which possess state-of-the-art performance in generative modeling, to predict LiDAR-like point clouds from paired raw radar data. We also incorporate the most recent diffusion model inference accelerating techniques to ensure that the proposed method can be implemented on MAVs with limited computing resources.We validate the proposed method through extensive benchmark comparisons and real-world experiments, demonstrating its superior performance and generalization ability. Code and pretrained models will be available at https://github.com/ZJU-FAST-Lab/Radar-Diffusion.
BibTeX
@article{zhang2024towards,
  author = {Ruibin Zhang and Donglai Xue and Yuhan Wang and Ruixu Geng and Fei Gao},
  title = {{Towards Dense and Accurate Radar Perception Via Efficient Cross-Modal Diffusion Model}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=3dDt96nYR3},
}
Figure