ROSS: Radar Off-Road Semantic Segmentation

Peng Jiang, Srikanth Saripalli

TL;DR

The ROSS (Radar Off-road Semantic Segmentation) paper introduces a pipeline to automatically generate training labels for radar data by transferring semantic information from LiDAR. By projecting 3D LiDAR-based labels onto 2D radar images, the researchers enable deep learning models to segment off-road environments even when visibility is poor—conditions where traditional cameras and LiDAR often fail.

What Problem Does This Paper Address?

Autonomous vehicles in off-road environments face a "fair-weather" problem. While LiDAR and cameras provide high-resolution data for semantic segmentation (identifying what is a rock, a bush, or a path), their performance plummets in rain, fog, or dust. RADAR is the logical alternative because its longer wavelengths penetrate these particles easily.

However, RADAR data is notoriously difficult to work with. It is noisy, has lower spatial resolution, and is primarily two-dimensional. Most importantly, manual labeling of raw radar data is nearly impossible for humans—it looks like a "snowy" mess of pixels compared to a clear camera image. This lack of annotated datasets has stalled the development of robust radar-based segmentation for off-road navigation.

Methods

The authors propose a cross-modal label transfer pipeline to bypass the need for manual radar labeling:

  1. LiDAR Semantic Mapping: They use a pre-trained model (Cylinder3D) on an existing off-road dataset (RELLIS-3D) to segment LiDAR scans.
  2. Odometry & Fusion: Using LiDAR-inertial odometry (FAST-LIO2), they aggregate multiple LiDAR scans into a dense, semantically labeled 3D point cloud map.
  3. Cross-Modal Projection: After precise extrinsic calibration between the sensors, they project the semantic labels from the 3D LiDAR map onto the 2D plane of the Radar data.
  4. Radar Representation: The radar data is represented as an image where the pixel value corresponds to the return energy.
  5. Deep Learning: They train a DeepLabv3 model using these generated labels. They experimented with both single-frame inputs and three-frame sequences to incorporate temporal context.

Datasets

  • RELLIS-3D: Used for initial training of the LiDAR segmentation model.
  • Custom Real-World Dataset: The authors collected data using a Clearpath Warthog robot equipped with a 128-channel Ouster LiDAR and a Navtech CR-DEV RADAR. This dataset includes 7 sequences totaling nearly 8,000 radar frames collected over three different days in off-road terrain.

Key Insights and Ideas

  • Intensity is Not Enough: The authors performed a "Class Intensity Distribution Analysis" and found that the return energy (brightness) of radar pixels for "Ground" and "Bushes" overlaps significantly. This means simple thresholding can't distinguish between a flat path and a thicket; deep learning is required to learn spatial patterns.
  • Temporal Context is a Game Changer: Moving from a single radar frame to a sequence of three frames (providing temporal/motion context) significantly improved the Intersection over Union (IoU) for difficult classes like "Bushes" (from 39% to 53%).
  • Class Consolidation: Radar lacks the resolution to distinguish between the 20+ classes found in LiDAR datasets. The authors found that grouping classes into broader categories—Ground, Bushes, and Obstacles—made the model much more reliable for navigation tasks.

Key Results

The researchers tested several model configurations (Ch1/Ch3 for input channels and Cls2/Cls3 for class groupings):

  • Three-Class Performance (Ch3Cls3): Achieved a mean IoU (mIoU) of 50.60% and a mean Accuracy (mAcc) of 66.95%.
  • Two-Class Performance: When simplifying the task to just "Ground vs. Everything Else," the model achieved stellar results, with accuracies reaching up to 97.8% for the Ground+Bushes category.
  • Reliability: The "Ground" class consistently showed the highest IoU, suggesting that radar is highly effective at identifying traversable surfaces in off-road conditions.

Limitations and Future Work

  • The "LiDAR Shadow" Problem: Because the radar labels are generated from LiDAR, the training data is limited by the LiDAR's range and line-of-sight. Radar can often "see" further or through some obstacles that LiDAR cannot; currently, those areas remain unlabelled.
  • 2D vs. 3D: The current approach projects everything onto a 2D image. The authors suggest that moving toward voxel-based representations or 3D radar point clouds could preserve more spatial nuance.
  • Radar-Centric Classes: Future work may involve defining classes based on how radar actually "sees" the world (e.g., density and slope) rather than just forcing LiDAR-based categories onto radar data.

Summary updated

Abstract

As the demand for autonomous navigation in off-road environments increases, the need for effective solutions to understand these surroundings becomes essential. In this study, we confront the inherent complexities of semantic segmentation in RADAR data for off-road scenarios. We present a novel pipeline that utilizes LIDAR data and an existing annotated off-road LIDAR dataset for generating RADAR labels, in which the RADAR data are represented as images. Validated with real-world datasets, our pragmatic approach underscores the potential of RADAR technology for navigation applications in off-road environments.
BibTeX
@inproceedings{jiang2023ross,
  author = {Peng Jiang and Srikanth Saripalli},
  title = {{ROSS: Radar Off-Road Semantic Segmentation}},
  booktitle = {Iser},
  year = {2023},
  doi = {10.1007/978-3-031-63596-0\_23},
}

Loading related content...

Figure