Activity Recognition Based on Millimeter-Wave Radar by Fusing Point Cloud and Range–Doppler Information

Yuchen Huang, Wei Li, Zhiyang Dou, Wantong Zou, Anye Zhang, Zan LiSignals 2022 · DOI

TL;DR

This paper introduces a novel deep learning approach for human activity recognition using millimeter-wave radar. By fusing point cloud and range-Doppler information, the method achieves a state-of-the-art accuracy of 97.26%, outperforming approaches that rely on a single data modality. This fusion strategy enhances the robustness and comprehensiveness of activity recognition, particularly in challenging environments.

What Problem Does This Paper Address?

Human activity recognition (HAR) is a critical area with applications ranging from healthcare to security. While various sensing technologies like LiDAR and computer vision have been explored, they often struggle in environments with poor visibility (smoke, fog, dust) or raise privacy concerns. Millimeter-wave (mmWave) radar offers a compelling alternative due to its penetration capabilities and inherent privacy-preserving nature. However, previous mmWave HAR research has typically focused on either point cloud data (spatial information) or range-Doppler data (temporal and velocity information) in isolation. This paper addresses the limitation of relying on a single data source by proposing a method to effectively fuse these two distinct types of mmWave radar data for more accurate and robust activity recognition.

Methods

The core of this paper is a multi-model deep learning architecture designed for feature fusion. The authors process 3D point cloud data using a 3D Convolutional Neural Network (CNN) combined with a Long Short-Term Memory (LSTM) network (3D CNN + LSTM) to capture both spatial and temporal dynamics. For the range-Doppler data, they employ a 3D CNN. The features extracted by these two sub-networks are then concatenated and fed into a fully connected layer for final classification. This parallel processing and fusion strategy aims to leverage the complementary information present in both data types.

Datasets

The authors collected a new dataset for activity recognition using a 3D millimeter-wave radar (IWR1843) from 17 volunteers. This dataset comprises six distinct activities: boxing, jumping, squatting, walking, circling, and high-knee lifting. Each activity was recorded for 20 seconds per person. The data was split into training (80%) and validation (20%) sets.

Key Insights and Ideas

  • Multimodal Fusion is Key: The study strongly suggests that combining different data modalities from mmWave radar significantly boosts activity recognition accuracy compared to using single modalities.
  • Complementary Information: Point cloud data captures the spatial arrangement and movement of body parts, while range-Doppler data provides insights into velocity and temporal evolution. Fusing them creates a richer representation of human actions.
  • CNN-LSTM for Point Clouds: The combination of 3D CNN for spatial feature extraction and LSTM for temporal sequence modeling proves effective for point cloud data.
  • 3D CNN for Range-Doppler: A 3D CNN is found to be suitable for processing the range-Doppler data, which can be viewed as a sequence of 2D "heatmaps" over time.
  • Network Depth Matters: Ablation studies on network depth indicate that there's an optimal number of layers; both too few and too many layers can degrade performance, with deeper networks potentially leading to overfitting.

Key Results

The proposed fusion method achieved a recognition accuracy of 97.26%. This is approximately 1% higher than the best-performing individual networks that used only point cloud data (96.59% with 3D CNN + LSTM) or only range-Doppler data (95.85% with 3D CNN). The fusion network also demonstrated a faster convergence speed and a lower loss value (0.088) compared to single-modality approaches. Confusion matrices revealed that while most activities were recognized with high accuracy, "squatting" and "circling" were slightly more challenging to distinguish.

Limitations and Future Work

The authors acknowledge that while fusing point cloud and range-Doppler data is beneficial, fusing even more data types (e.g., echo intensity) could potentially lead to even more comprehensive activity descriptions and higher accuracy. They suggest that future work could explore expanding the types of input data, optimizing feature extraction processes, and developing more sophisticated methods for integrating diverse data streams to further enhance network performance. The study also notes that increasing network depth beyond a certain point can lead to overfitting, suggesting a need for careful architecture design.

Summary updated

Abstract

Millimeter-wave radar has demonstrated its high efficiency in complex environments in recent years, which outperforms LiDAR and computer vision in human activity recognition in the presence of smoke, fog, and dust. In previous studies, researchers mostly analyzed either 2D (3D) point cloud or range–Doppler information from radar echo to extract activity features. In this paper, we propose a multi-model deep learning approach to fuse the features of both point clouds and range–Doppler for classifying six activities, i.e., boxing, jumping, squatting, walking, circling, and high-knee lifting, based on a millimeter-wave radar. We adopt a CNN–LSTM model to extract the time-serial features from point clouds and a CNN model to obtain the features from range–Doppler. Then we fuse the two features and input the fused feature into the full connected layer for classification. We built a dataset based on a 3D millimeter-wave radar from 17 volunteers. The evaluation result based on the dataset shows that this method has higher accuracy than utilizing the two kinds of information separately and achieves a recognition accuracy of 97.26%, which is about 1% higher than other networks with only one kind of data as input.
BibTeX
@inproceedings{huang2022activity,
  author = {Yuchen Huang and Wei Li and Zhiyang Dou and Wantong Zou and Anye Zhang and Zan Li},
  title = {{Activity Recognition Based on Millimeter-Wave Radar by Fusing Point Cloud and Range–Doppler Information}},
  booktitle = {Signals},
  year = {2022},
  doi = {10.3390/signals3020017},
}
Figure