Towards Dense and Accurate Radar Perception via Efficient Cross-Modal Diffusion Model

Ruibin Zhang, Donglai Xue, Yuhan Wang, Ruixu Geng, Fei Gao

TL;DR

This paper introduces a novel approach to transform sparse, noisy mmWave radar data into dense, LiDAR-like point clouds using a cross-modal diffusion model. By leveraging Consistency Models for distillation, the authors achieve a massive 80x inference speedup, enabling high-quality, one-step environmental perception suitable for real-time navigation on resource-constrained micro aerial vehicles (MAVs).

What Problem Does This Paper Address?

Millimeter-wave (mmWave) radars are the "all-weather" heroes of robotics; unlike LiDAR or cameras, they can see through fog, smoke, and dust. However, they have a major drawback: their output is incredibly sparse and noisy. Traditional signal processing (like CFAR) produces "point clouds" that are often too thin for a drone to use for complex obstacle avoidance or SLAM (Simultaneous Localization and Mapping).

While deep learning has been used to "upsample" radar data before, previous methods (like GANs or standard U-Nets) often struggle to generate sharp structural details or fail to generalize to new environments. Furthermore, high-performing generative models like Diffusion Models are typically too slow for the split-second decision-making required by a flying drone.

Methods

The authors frame the radar-to-LiDAR task as an image restoration problem.

  1. Data Representation: They convert raw radar signals into Range-Azimuth Heatmaps (RAH) and LiDAR data into Bird’s Eye View (BEV) images.
  2. Diffusion Framework: They use a diffusion model (based on the EDM framework) to learn the mapping from the "impaired" radar domain to the "original" LiDAR domain.
  3. Loss Functions: To ensure the generated maps aren't just accurate on a pixel level but also look "right" to a robot, they combine standard Mean Squared Error (MSE) with LPIPS (Learned Perceptual Image Patch Similarity). This helps the model prioritize structural features like straight walls over unimportant textures like grass.
  4. Efficiency via Consistency Models: To solve the "slow inference" problem of diffusion, they distill their trained model into a Consistency Model. This allows the network to jump from pure noise to a final prediction in a single step, rather than iterating dozens of times.

Datasets

  • ColoRadar: A public dataset containing 52 sequences of synchronized mmWave radar and LiDAR data across diverse environments (labs, hallways, mines, and outdoors).
  • Self-Made Dataset: The authors collected 3,152 frames of data at the Huzhou Institute of Zhejiang University using a custom handheld rig (Nvidia Jetson Orin NX, Livox Mid-360 LiDAR, and a TI-AWR1843BOOST radar) to test how well the model generalizes to entirely unseen sensors and scenes.

Key Insights and Ideas

  • Diffusion as a Denoising Expert: The authors argue that because mmWave signals are inherently noisy and low-resolution, the "denoising" nature of diffusion models is a perfect mathematical fit for this specific sensor-fusion task.
  • The "One-Step" Breakthrough: The most significant hurdle for diffusion in robotics is latency. By using consistency distillation, they prove you can maintain generative quality while achieving the speeds necessary for a robot moving in real-time.
  • Cross-Modal Supervision: The model doesn't just "clean" radar; it learns to "hallucinate" the density of LiDAR by using it as a ground-truth supervisor during training. This allows a cheap radar sensor to mimic a much more expensive LiDAR sensor.

Key Results

  • Superior Accuracy: The proposed method outperformed baselines (OS-CFAR, RadarHD, RPDNet) across almost all metrics, including Chamfer Distance (CD) and Hausdorff Distance (HD).
  • 80x Speedup: The Consistency Model achieved an 80x acceleration compared to the iterative EDM version without significant loss in visual quality.
  • Real-Time Performance: On an embedded Nvidia Jetson Orin NX, the model achieved an inference time of 181.6 milliseconds (~5 Hz), which is sufficient for MAV autonomous navigation.
  • Generalization: The model trained on the public ColoRadar dataset worked successfully on the authors' custom hardware in a completely different environment, proving it isn't just "memorizing" specific lab layouts.

Limitations and Future Work

  • 2D vs. 3D: Currently, the model focuses on 2D Bird's Eye View (BEV) mapping, omitting elevation data. The authors plan to explore 3D point cloud generation in future iterations.
  • Low-Reflectivity Objects: In the "Longboard" (outdoor) scene, the model struggled with objects like grass and leaves. These materials have very low radar reflectivity and are highly unstructured, making them difficult for the neural network to "learn" and reconstruct.
  • Dynamic Objects: The paper primarily focuses on static scene mapping; handling high-speed dynamic obstacles remains a challenge for future research.

Summary updated

Abstract

Millimeter wave (mmWave) radars have attracted significant attention from both academia and industry due to their capability to operate in extreme weather conditions. However, they face challenges in terms of sparsity and noise interference, which hinder their application in the field of micro aerial vehicle (MAV) autonomous navigation. To this end, this letter proposes a novel approach to dense and accurate mmWave radar point cloud construction via cross-modal learning. Specifically, we introduce diffusion models, which possess state-of-the-art performance in generative modeling, to predict LiDAR-like point clouds from paired raw radar data. We also incorporate the most recent diffusion model inference accelerating techniques to ensure that the proposed method can be implemented on MAVs. We validate the proposed method through extensive benchmark comparisons and real-world experiments, demonstrating its superior performance and generalization ability.
BibTeX
@article{zhang2024towards,
  author = {Ruibin Zhang and Donglai Xue and Yuhan Wang and Ruixu Geng and Fei Gao},
  title = {{Towards Dense and Accurate Radar Perception via Efficient Cross-Modal Diffusion Model}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=3dDt96nYR3},
}

Loading related content...

Figure