Dense 3D Point Cloud Environmental Mapping Using Millimeter-Wave Radar

Zhiyuan Zeng, Jie Wen, Jianan Luo, Gege Ding, Xiongfei GengSensors 2024 · DOI

TL;DR

This paper introduces a novel method for creating dense, high-resolution 3D environmental maps using millimeter-wave (mmWave) radar. By combining radar SLAM for robust submap generation with a cross-modal deep learning network trained on LiDAR data, they achieve a significant increase in point cloud density and accuracy, enabling more detailed environmental perception for applications like autonomous driving.

What Problem Does This Paper Address?

Current millimeter-wave radar systems, while offering advantages like long range and penetration through adverse weather, suffer from sparse 3D point clouds. This sparsity limits their ability to create detailed environmental maps, hindering applications that require high-fidelity perception. Existing methods struggle with poor angular resolution, noise, and the dynamic nature of real-world scenarios, especially those involving vehicles with varying speeds. The paper aims to bridge this gap by generating dense, accurate 3D mmWave radar maps that rival the quality of LiDAR.

Methods

The proposed method consists of three main stages:

  1. Radar Submap Generation: Instead of directly using raw radar point cloud frames, the authors employ a radar SLAM (Simultaneous Localization and Mapping) approach. Specifically, they use a Doppler odometry algorithm to create local "submaps" from multiple radar frames. This process helps to adapt to varying vehicle speeds and reduces data dimensionality, making it more suitable for subsequent deep learning.
  2. Point Cloud Projection and Normalization: To address the inherent differences in resolution and sensing mechanisms between mmWave radar and LiDAR, a normalization process is introduced. This involves projecting both LiDAR and radar point clouds into a depth image representation. This step aims to reduce LiDAR's resolution to match the radar's and to filter out spurious points caused by radar diffraction.
  3. 3D-RadarHR Network: A U-shaped sparse convolutional neural network, termed 3D-RadarHR, is proposed. This network takes the normalized depth images from radar submaps as input and is trained using LiDAR depth images as the ground truth. The goal is to learn a transformation that generates high-resolution mmWave radar depth images, effectively densifying the radar point clouds.

Datasets

The paper utilizes a dataset collected from a vehicle equipped with mmWave radar, LiDAR, camera, GNSS, and IMU. This dataset comprises approximately 126,180 valid data samples, with 40% from outdoor environments and 60% from subterranean tunnels. The data was split into training, validation/development, and testing sets. LiDAR data serves as the ground truth for training the 3D-RadarHR network.

Key Insights and Ideas

  • Radar SLAM for Robustness: Using radar SLAM to create submaps is a crucial step. It not only densifies the point cloud but also makes the system more robust to varying vehicle speeds, a common challenge in autonomous driving.
  • Cross-Modal Learning with Normalization: The explicit normalization step between radar and LiDAR data is key to enabling effective deep learning. By aligning their representations, the network can learn to translate the sparse radar data into a dense map, leveraging the rich information from LiDAR.
  • Semantic Information for Enhanced Mapping: Integrating semantic information (though not fully detailed in the provided sections, it's mentioned as a contribution) can further improve map quality by providing context and aiding in noise suppression.
  • Trade-off in Submap Density: There's an optimal "Density Growth Factor" for radar submaps. Increasing it too much can lead to accumulated odometry errors, while too little doesn't provide sufficient density. The paper identifies a sweet spot around 18 for their experiments.

Key Results

  • Density Increase: The proposed method achieves over a 50-fold increase in point cloud density for mmWave radar environmental maps.
  • Accuracy: The reconstructed point cloud accuracy is better than 0.1 m.
  • Performance: The method achieves superior environmental map reconstruction performance compared to existing algorithms.
  • Real-time Capability: The system maintains a real-time processing rate of 15 Hz.
  • Qualitative Improvements: Experimental results show significantly denser and more detailed maps, with effective noise and interference suppression, especially in challenging scenarios like tunnels and during sudden vehicle maneuvers.

Limitations and Future Work

  • Algorithm Practicality: Current methods are often limited to specific targets or small indoor scenes, not fully meeting the demands of large outdoor environments.
  • Algorithm Robustness: Adapting to dynamic, real-time scenarios with non-uniform motion remains a challenge. Deep learning networks often assume uniform motion, which can lead to blurred reconstructions.
  • Data Acquisition: Radar datasets are more challenging and less stable to acquire than visual datasets, impacting the robustness of deep learning methods.
  • Future Work: The authors suggest further research into incorporating Doppler information more effectively, designing dedicated radar point cloud pose estimation networks, and collecting larger datasets that capture non-uniform motion. They also mention exploring applications like millimeter-wave radar environmental map recognition.

Summary updated

Abstract

To address the challenges of sparse point clouds in current MIMO millimeter-wave radar environmental mapping, this paper proposes a dense 3D millimeter-wave radar point cloud environmental mapping algorithm. In the preprocessing phase, a radar SLAM-based approach is introduced to construct local submaps, which replaces the direct use of radar point cloud frames. This not only reduces data dimensionality but also enables the proposed method to handle scenarios involving vehicle motion with varying speeds. Building on this, a 3D-RadarHR cross-modal learning network is proposed, which uses LiDAR as the target output to train the radar submaps, thereby generating a dense millimeter-wave radar point cloud map. Experimental results across multiple scenarios, including outdoor environments and underground tunnels, demonstrate that the proposed method can increase the point cloud density of millimeter-wave radar environmental maps by over 50 times, with a point cloud accuracy better than 0.1 m. Compared to existing algorithms, the proposed method achieves superior environmental map reconstruction performance while maintaining a real-time processing rate of 15 Hz.
BibTeX
@inproceedings{zeng2024dense,
  author = {Zhiyuan Zeng and Jie Wen and Jianan Luo and Gege Ding and Xiongfei Geng},
  title = {{Dense 3D Point Cloud Environmental Mapping Using Millimeter-Wave Radar}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24206569},
}
Figure