ROSS: Radar Off-road Semantic Segmentation

Peng Jiang, Srikanth Saripalli

Abstract

As the demand for autonomous navigation in off-road environments increases, the need for effective solutions to understand these surroundings becomes essential. In this study, we confront the inherent complexities of semantic segmentation in RADAR data for off-road scenarios. We present a novel pipeline that utilizes LIDAR data and an existing annotated off-road LIDAR dataset for generating RADAR labels, in which the RADAR data are represented as images. Validated with real-world datasets, our pragmatic approach underscores the potential of RADAR technology for navigation applications in off-road environments.

Introduction

Autonomous vehicles operating in off-road environments frequently encounter a "fair-weather" limitation. While LiDAR and cameras provide high-resolution data for semantic segmentation—identifying terrain features like rocks, bushes, or paths—their performance degrades significantly in adverse conditions such as rain, fog, or dust. RADAR offers a robust alternative due to its longer wavelengths, which penetrate these atmospheric particles with ease.

However, RADAR data presents unique challenges: it is noisy, has lower spatial resolution, and is primarily two-dimensional. Crucially, manual labeling of raw radar data is nearly impossible for human annotators, as it often appears as a chaotic "snowy" pattern. This lack of annotated datasets has hindered the development of radar-based segmentation. The ROSS (Radar Off-road Semantic Segmentation) pipeline addresses this by automatically generating training labels through the transfer of semantic information from LiDAR.

Methodology

The proposed pipeline bypasses the need for manual radar labeling by projecting 3D LiDAR-based labels onto 2D radar images. The process involves five key stages:

  1. LiDAR Semantic Mapping: A pre-trained Cylinder3D model, trained on the RELLIS-3D off-road dataset, segments incoming LiDAR scans.
  2. Odometry & Fusion: Using FAST-LIO2 (LiDAR-inertial odometry), multiple scans are aggregated into a dense, semantically labeled 3D point cloud map.
  3. Cross-Modal Projection: Precise extrinsic calibration allows the projection of semantic labels from the 3D LiDAR map onto the 2D radar plane.
  4. Radar Representation: Radar data is processed into an image format where pixel values correspond to return energy.
  5. Deep Learning: A DeepLabv3 model is trained on these generated labels, utilizing both single-frame and multi-frame (temporal) inputs.
The ROSS pipeline for cross-modal label transfer from LiDAR to RADAR.

Experimental Setup

The researchers utilized two primary data sources to validate the pipeline:

  • RELLIS-3D: An existing dataset used for the initial training of the LiDAR segmentation component.
  • Custom Real-World Dataset: Data was collected using a Clearpath Warthog robot equipped with a 128-channel Ouster LiDAR and a Navtech CR-DEV RADAR. This collection includes nearly 8,000 radar frames across seven sequences in diverse off-road terrain.

Results and Discussion

Analysis of the radar data revealed that return energy (intensity) alone is insufficient for classification; "Ground" and "Bushes" show significant overlap in intensity distributions. This necessitates deep learning to identify complex spatial patterns.

A key finding was the impact of temporal context. Transitioning from a single radar frame to a three-frame sequence improved the Intersection over Union (IoU) for the difficult "Bushes" class from 39% to 53%. Furthermore, consolidating the 20+ LiDAR classes into broader categories—Ground, Bushes, and Obstacles—significantly improved model reliability for navigation tasks.

Radar data representation and semantic segmentation results.

The best-performing configuration (three-frame input, three classes) achieved a mean IoU (mIoU) of 50.60% and a mean Accuracy (mAcc) of 66.95%. For binary "Ground vs. Everything Else" tasks, accuracies reached as high as 97.8%.

Conclusion and Future Work

While ROSS demonstrates the viability of radar for off-road segmentation, it faces the "LiDAR Shadow" problem: training data is constrained by the LiDAR's line-of-sight, even though radar can often see further or through certain obstacles. Future research will explore voxel-based representations to preserve 3D spatial nuance and define radar-centric classes based on material density and slope rather than purely visual LiDAR categories.

Summary updated

Abstract

As the demand for autonomous navigation in off-road environments increases, the need for effective solutions to understand these surroundings becomes essential. In this study, we confront the inherent complexities of semantic segmentation in RADAR data for off-road scenarios. We present a novel pipeline that utilizes LIDAR data and an existing annotated off-road LIDAR dataset for generating RADAR labels, in which the RADAR data are represented as images. Validated with real-world datasets, our pragmatic approach underscores the potential of RADAR technology for navigation applications in off-road environments.
BibTeX
@inproceedings{jiang2023ross,
  author = {Peng Jiang and Srikanth Saripalli},
  title = {{ROSS: Radar Off-road Semantic Segmentation}},
  booktitle = {Iser},
  year = {2023},
  doi = {10.1007/978-3-031-63596-0\_23},
}
Figure