HDNet: Hierarchical Dynamic Network for Gait Recognition using Millimeter-wave radar

Yu Fu, Yanyan Huang, Yong Wang, Kun Shi, Chaojie Gu, Yu Fu 0008, Cheng Zhuo, Zhiguo Shi

TL;DR

This paper introduces HDNet, a novel hierarchical network for gait recognition using millimeter-wave (mmWave) radar. HDNet leverages a new "point flow" descriptor to capture dynamic inter-frame information and a dynamic frame sampling module to improve efficiency, achieving state-of-the-art results on mmWave gait recognition benchmarks.

What Problem Does This Paper Address?

Gait recognition, the process of identifying individuals by their walking style, has many applications, from personalized services to surveillance. While RGB cameras are common for this task, they struggle in poor lighting and raise privacy concerns. Millimeter-wave (mmWave) radar offers a promising alternative due to its robustness, non-invasiveness, and privacy-preserving nature. However, existing mmWave gait recognition methods primarily focus on aggregating spatial-temporal features within frames, neglecting the crucial dynamic information that exists between frames, which is essential for understanding the periodic nature of human walking. This paper aims to bridge this gap by developing a method that effectively captures and utilizes these inter-frame dynamics.

Methods

The proposed HDNet architecture comprises three main components:

  • Spatial GNN Backbone: This module uses a graph neural network (specifically AdaptConv) to extract spatial features from individual radar point cloud frames. It's designed to capture relationships between points from different parts of the human body.
  • Dynamic Frame Sampling (DFS): To improve computational efficiency and focus on informative frames, HDNet introduces a dynamic frame sampling module. This module learns to predict which frames are most relevant for gait recognition and prunes less informative ones. It uses a Gumbel-Softmax reparameterization technique to enable end-to-end training of this sampling process. A "mask loss" is also introduced to constrain the ratio of sampled frames.
  • Temporal Aggregator (TA): This component utilizes a Transformer-based architecture to aggregate the extracted spatial features and the novel "point flow" features across the sampled frames. Transformers are chosen for their ability to model long-range dependencies, which is well-suited for the periodic nature of gait.

A key novelty is the point flow descriptor. This descriptor captures the dynamic changes between consecutive radar point cloud frames by calculating the difference in Doppler velocity between corresponding points in adjacent frames. This provides a new dimension of information about the walking motion.

Datasets

The authors evaluate HDNet on two public mmWave radar-based gait recognition datasets:

  • mmGait [3]: A large dataset containing 30 hours of 3D point cloud data from 95 volunteers, collected using TI IWR1443 and TI IWR6843 mmWave sensing devices. It includes both fixed and free-route walking trajectories.
  • STPointGCN [5]: A dataset with 75 minutes of 3D point cloud data from 6 participants, collected in three different environments using TI IWR6843.

For experiments, the authors focus on scenarios with a single person walking to simplify the problem and ensure fair comparison with other methods.

Key Insights and Ideas

  • Point Flow for Dynamics: The introduction of "point flow" as a feature descriptor is a significant contribution. It effectively captures the subtle, dynamic changes between frames that are crucial for gait recognition, moving beyond static spatial-temporal feature aggregation.
  • Dynamic Frame Sampling for Efficiency: The DFS module is a clever way to reduce computational load without sacrificing accuracy. By adaptively selecting informative frames, it makes the model more efficient, especially for longer sequences.
  • Transformer's Suitability for Gait: The success of the Transformer-based Temporal Aggregator highlights its power in modeling the periodic and long-range dependencies inherent in human gait.
  • mmWave Radar's Potential: The paper further solidifies mmWave radar as a viable and powerful sensor for gait recognition, demonstrating its ability to achieve high accuracy in a privacy-preserving manner.

Key Results

HDNet demonstrates superior performance on both datasets:

  • On the mmGait dataset, HDNet achieves an accuracy of 96.52%, outperforming the previous state-of-the-art model SRPNet by 13.7% in accuracy.
  • On the STPointGCN dataset, HDNet achieves an accuracy of 91.40%, outperforming the SOTA model STPointGCN by 21% in accuracy.
  • The Dynamic Frame Sampling module shows significant benefits. Even when the sampling ratio is reduced to 0.3, HDNet with DFS maintains an accuracy above 90%, significantly outperforming random sampling at the same ratio.

Limitations and Future Work

  • Single-Person Focus: The current evaluation primarily uses data with only one person walking. While this simplifies comparison, real-world scenarios often involve multiple people. Future work could explore extending HDNet to handle multi-target scenarios more robustly.
  • Dataset Size: While the datasets are substantial, the authors acknowledge the need for more diverse and larger datasets to further validate the model's generalization capabilities.
  • Computational Cost: Although DFS improves efficiency, the overall computational cost of HDNet might still be a consideration for real-time, resource-constrained applications. Further optimization could be explored.

Summary updated

Abstract

Gait recognition is widely used in diversified practical applications. Currently, the most prevalent approach is to recognize human gait from RGB images, owing to the progress of computer vision technologies. Nevertheless, the perception capability of RGB cameras deteriorates in rough circumstances, and visual surveillance may cause privacy invasion. Due to the robustness and non-invasive feature of millimeter wave (mmWave) radar, radar-based gait recognition has attracted increasing attention in recent years. In this research, we propose a Hierarchical Dynamic Network (HDNet) for gait recognition using mmWave radar. In order to explore more dynamic information, we propose point flow as a novel point clouds descriptor. We also devise a dynamic frame sampling module to promote the efficiency of computation without deteriorating performance noticeably. To prove the superiority of our methods, we perform extensive experiments on two public mmWave radar-based gait recognition datasets, and the results demonstrate that our model is superior to existing state-of-the-art methods.
BibTeX
@inproceedings{fu2023hdnet,
  author = {Yu Fu and Yanyan Huang and Yong Wang and Kun Shi and Chaojie Gu and Yu Fu and Cheng Zhuo and Zhiguo Shi},
  title = {{HDNet: Hierarchical Dynamic Network for Gait Recognition using Millimeter-wave radar}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096835},
}
Figure