Human Detection from 4D Radar Data in Low-Visibility Field Conditions

Mikael Skog, Oleksandr Kotlyar, Vladimír Kubelka, Martin MagnussonarXiv 2024 · DOI · arXiv

Introduction

Reliable human detection is a critical safety requirement for autonomous vehicles and industrial robotics, particularly in high-risk environments like underground mines or construction sites. While cameras and lidars are standard for perception, they suffer from significant performance degradation in "adverse field conditions"—environments filled with thick dust, smoke, or water mist. In these scenarios, airborne particulates scatter light and laser pulses, effectively blinding traditional sensors.

The TMVA4D Architecture

To address these challenges, researchers developed TMVA4D (Temporal Multi-View Architecture for 4D), a convolutional neural network (CNN) specifically designed for 4D imaging radar data. Unlike traditional 2D radar, 4D radar provides elevation information and Doppler velocity, which are essential for distinguishing humans from ground clutter or detecting fallen individuals.

The TMVA4D model employs several key technical strategies:

  • Multi-View Projections: To manage the computational complexity of 4D tensors, the model projects radar data into five 2D heatmaps: Elevation-Azimuth (EA), Elevation-Range (ER), Elevation-Doppler (ED), Range-Azimuth (RA), and Doppler-Azimuth (DA).
  • Temporal Stacking: The network processes a sequence of five consecutive frames (the current frame plus four previous ones) to mitigate radar noise and improve detection consistency over time.
  • Encoder-Decoder with ASPP: The architecture uses dedicated encoders for each view, fused into a latent space, and a decoder with Atrous Spatial Pyramid Pooling (ASPP) to capture multi-scale features, outputting a semantic mask in the camera-like Elevation-Azimuth view.
Radar points overlaid on thermal imagery for ground truth validation.

Dataset and Training

The study utilized a novel dataset of 102,966 frames collected in challenging industrial settings, including underground mines and machinery car washes. Because manual labeling of obscured camera footage is impossible, the researchers used thermal imaging as a reference. A pre-trained YOLOv8 model was applied to the thermal data to generate high-quality semantic masks, which served as the ground truth for training the radar-based model.

Results and Discussion

The TMVA4D model demonstrated robust performance, achieving a Mean Intersection over Union (mIoU) of 78.2% and a Mean Dice score of 86.1% for human detection. Crucially, the system maintains real-time performance with an average processing time of under 50ms per frame on standard hardware (NVIDIA RTX 2080 Ti).

Comparison of sensor performance in thick mist: (a) color camera, (b) thermal camera, (c) 4D radar detections, and (d) the resulting semantic mask.

As shown in the comparison above, while the color camera is completely obscured by mist, the 4D radar successfully identifies human targets. The inclusion of the elevation dimension proved vital; without it, low-profile targets like fallen persons are nearly indistinguishable from the environment.

Limitations and Future Work

Despite its success, the model shows sensitivity to human pose, with lower accuracy for people lying down compared to those walking. Performance also decreases when the sensor platform is in motion due to increased Doppler complexity. Future research will focus on improving robustness to ego-motion and expanding the dataset to include a wider variety of industrial obstacles and human poses.

Summary updated

BibTeX
@misc{skog2024human,
  author = {Mikael Skog and Oleksandr Kotlyar and Vladimír Kubelka and Martin Magnusson},
  title = {{Human Detection from 4D Radar Data in Low-Visibility Field Conditions}},
  year = {2024},
  eprint = {2404.05307},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.05307},
}
Figure