TL;DR
This paper introduces RaSS, a novel framework for semantic segmentation of 4D mm-Wave radar point clouds, tackling the challenges of sparsity and noise. By leveraging knowledge distillation from a LiDAR-based teacher model and incorporating an adaptive Doppler compensation module, RaSS significantly improves radar-only semantic segmentation performance, paving the way for more robust autonomous driving perception.
What Problem Does This Paper Address?
Autonomous driving systems rely heavily on environmental perception to navigate safely. While LiDAR and cameras are common sensors, they struggle in adverse weather conditions like fog or heavy rain. 4D mm-Wave radar offers a solution, providing 3D point clouds with Doppler velocity information, which works well in challenging weather. However, radar data is inherently sparse and noisy, making point-level tasks like semantic segmentation (labeling every point in the scene) difficult. Existing radar perception research has largely focused on object detection (bounding boxes), leaving semantic segmentation of radar point clouds underexplored. This paper aims to bridge this gap by developing methods for accurate semantic segmentation using 4D radar data.
Methods
The core of the proposed solution is the RaSS (Radar Semantic Segmentation) framework, which employs a cross-modal knowledge distillation approach. Here's how it works:
- Teacher-Student Architecture: A pre-trained LiDAR-based semantic segmentation model acts as the "teacher," providing rich, high-quality features. A radar-only model serves as the "student," learning from the teacher.
- Adaptive Doppler Compensation (ADC): A key innovation is the ADC module. Since radar Doppler speed is relative to the ego-vehicle's motion, static objects can appear to have velocity. ADC estimates and compensates for this ego-motion-induced velocity, providing a cleaner velocity feature for the student model. This is particularly useful as it works even with unknown ego-vehicle speeds.
- Radar Feature Knowledge Distillation (RFKD): To transfer knowledge effectively despite the data differences, RFKD is used. It aggregates features from both LiDAR and radar point clouds into a bird's-eye-view (BEV) representation. Crucially, it then selects spatially aligned features from both modalities to perform distillation. This selective distillation helps the radar student model learn discriminative features that mimic the LiDAR teacher's understanding of the scene structure.
- Backbone: The paper uses standard 3D semantic segmentation backbones like Cylinder3D and PT-V2 for both the teacher and student models, allowing for direct feature comparison and distillation.
Datasets
The authors introduce a new benchmark dataset:
- ZJUSSet: This dataset comprises synchronized 4D mm-Wave radar, LiDAR, and camera data collected in urban environments. It features accurate point-wise annotations for up to 10 semantic categories. The dataset is designed to specifically address the challenges of 4D radar semantic segmentation, offering a more comprehensive annotation than previous radar datasets.
- VoD Dataset: The proposed RaSS framework is also evaluated on the View of Delft (VoD) dataset. While VoD is primarily for 3D object detection, the authors generated pseudo-semantic segmentation labels from its ground truth bounding boxes to validate their method's generalizability.
Key Insights and Ideas
- 4D Radar's Untapped Potential: The paper strongly suggests that 4D radar, despite its sparsity and noise, holds significant potential for point-level semantic segmentation, a task previously thought to be beyond its capabilities.
- Doppler Compensation is Crucial: The Adaptive Doppler Compensation module is shown to be vital for improving radar-only segmentation. By accurately accounting for ego-motion, it provides a more reliable velocity signal, which is a key differentiator for radar.
- Selective Knowledge Distillation Matters: Simply distilling all features isn't optimal. The RFKD's approach of aligning and selecting spatially corresponding features between LiDAR and radar BEV representations is key to effectively transferring knowledge and overcoming domain gaps.
- Cross-Modal Distillation as a Bridge: Knowledge distillation from a dense, accurate sensor like LiDAR to a sparse, noisy sensor like radar is a powerful paradigm for improving the performance of the latter.
Key Results
- ZJUSSet Performance: The RaSS framework, using the PT-V2 backbone, achieved an mIoU of 42.35% on the ZJUSSet validation set. This is a significant improvement over existing radar-only methods, outperforming baselines by 5.57% mIoU.
- LiDAR vs. Radar Gap: While RaSS significantly closes the gap, LiDAR-based methods still achieve higher mIoU (around 69-72%), highlighting the inherent challenges of radar data.
- Real-time Performance: The RaSS model meets real-time requirements for autonomous driving, with inference times of 74 ms (Cylinder3D) and 60 ms (PT-V2), which are faster than their LiDAR counterparts.
- Memory Efficiency: Radar sensors offer substantially lower memory usage compared to LiDAR.
- VoD Dataset: On the VoD dataset, RaSS also outperformed baselines, demonstrating its generalizability.
Limitations and Future Work
- Performance Gap with LiDAR: The paper acknowledges that a performance gap still exists between radar-only segmentation (RaSS) and LiDAR-based methods. This is attributed to the inherent sparsity and noise in radar data.
- Minority Categories: The failure case analysis shows that the model struggles with minority categories (e.g., cyclist, tricycle) and small objects, likely due to insufficient training data for these classes.
- Future Work: The authors plan to:
- Augment the ZJUSSet with more data, especially under adverse weather conditions.
- Develop more advanced algorithms to further improve semantic segmentation performance.
- Explore more potential applications of 4D mm-wave radar in autonomous driving.
Summary updated