Non-Contact Cross-Person Activity Recognition by Deep Metric Ensemble Learning

Chen Ye, Siyuan Xu, Zhengran He, Yue Yin, Tomoaki Ohtsuki, Guan GuiBioengineering 2024 · DOI

TL;DR

This paper introduces SE-ABLSTM-C, a novel Wi-Fi-based system for recognizing human activities performed by individuals not seen during training (cross-person activity recognition or CPAR). By combining snapshot ensemble learning with a center loss metric learning approach, SE-ABLSTM-C significantly improves generalization capabilities, achieving application-level accuracy for recognizing seven distinct activities.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) is crucial for applications like elderly monitoring and intrusion detection. While Wi-Fi-based systems offer non-contact and privacy-preserving advantages, existing deep learning methods often struggle with cross-person activity recognition (CPAR). This means they perform well when recognizing activities of people they were trained on, but fail when encountering new individuals whose activity patterns might differ. This lack of generalization limits their real-world applicability, as retraining for every new user is impractical. The paper aims to bridge this gap by developing a method that can accurately recognize activities of unknown individuals without requiring prior training data from them.

Methods

The proposed method, SE-ABLSTM-C, builds upon the Attention-based Bi-directional Long Short-Term Memory (ABLSTM) network, a state-of-the-art DNN for HAR. The novelty lies in two key components:

  • Snapshot Ensemble Learning: Instead of training a single model, this technique trains multiple "base-classifiers" within a single training process. It leverages a cyclic cosine annealing learning rate schedule to save model weights at different points (local minima) during training. This creates a diverse set of models that, when combined, enhance generalization.
  • Metric Learning with Center Loss: To further improve the ability to distinguish between activities, especially for unseen individuals, the researchers incorporate a center loss function. This metric learning approach encourages features from the same activity class to be clustered closely together in the feature space, while pushing features from different classes further apart. This makes the learned representations more discriminative.

The SE-ABLSTM-C method combines these two techniques, using the hybrid loss function (softmax loss + center loss) to train the ABLSTM network within the snapshot ensemble framework.

Datasets

The study utilizes a newly collected CSI dataset for training and evaluation.

  • Our Dataset: This dataset was collected using ESP32 development boards and comprises Channel State Information (CSI) data from five male subjects performing seven activities (waving, clapping, walking, lying down, sitting down, falling, and picking up). The dataset contains 700 data points in total, with each subject repeating 20 rounds of the 7 activities. This is described as a "medium-scale" dataset.

The paper also references existing CSI datasets for HAR in Table 1, but these were not used for training or evaluation in this specific study.

Key Insights and Ideas

  • Generalization is Key for Real-World CPAR: The core challenge addressed is moving beyond recognizing activities of known individuals to recognizing activities of unknown individuals. This highlights the critical need for robust generalization in HAR systems.
  • Ensemble Learning for Diversity: Snapshot ensemble learning is a clever way to generate multiple diverse models from a single training run, effectively exploring different regions of the hypothesis space and improving robustness.
  • Metric Learning for Discriminative Features: The center loss is crucial for making the learned features more distinct. This is particularly important for CPAR, where subtle variations in activity patterns between individuals can be challenging to capture.
  • Wi-Fi CSI as a Promising Modality: The paper reinforces the potential of Wi-Fi CSI for non-contact, privacy-preserving activity recognition, especially in indoor environments.

Key Results

The proposed SE-ABLSTM-C method demonstrated significant improvements in cross-person activity recognition (CPAR) across five different training/testing patterns.

  • Task I (Seen Persons): As a baseline, conventional HAR methods were evaluated. ABLSTM achieved the highest average accuracy of 97.23% for recognizing activities of seen persons.
  • Task II (Unknown Persons - CPAR):
    • The SE-ABLSTM-C method markedly improved recognition accuracies compared to conventional HAR methods (CNN, LSTM, ABLSTM) and even its own components (SE-ABLSTM and ABLSTM-C).
  • For example, in the Pattern BCDE-A, the proposed SE-ABLSTM-C achieved an average accuracy of 86.68%, outperforming ABLSTM (83.57%), LSTM (83.29%), CNN (83.99%), and LAGMAT (85.67%).
  • Across all five CPAR patterns, SE-ABLSTM-C achieved the highest average accuracies, ranging from 78.86% to 89.03%.
  • The paper states that SE-ABLSTM-C improved the average accuracy of seven activities by approximately 3–14% compared to the ABLSTM method.

Limitations and Future Work

  • Falling Activity Recognition: The paper acknowledges that the performance for recognizing the "falling" activity remains relatively unsatisfied, even with the proposed method. This is attributed to the similarity between falling and picking up, making them difficult to distinguish.
  • Dataset Scale: The authors note that their dataset is of medium scale, and increasing the number of subjects could further enhance the study.
  • Activity Similarity: The vulnerability of Wi-Fi-based recognition to similar activities is a general limitation.
  • Future Work:
    • Investigating CPAR for activities with similar categories.
  • Addressing open-set problems where activities unknown during training might appear.
  • Exploring parallel computing with Transformer modules for efficiency.
  • Integrating Wi-Fi sensing with camera data for human gait trajectory generation.
  • The authors suggest that in scenarios requiring robust recognition of certain activities, similar categories like falling and picking up might need to be excluded.

Summary updated

Abstract

In elderly monitoring or indoor intrusion detection, the recognition of human activity is a key task. Owing to several strengths of Wi-Fi-based devices, including their non-contact and privacy protection, these devices have been widely applied in the area of smart homes. By the deep learning technique, numerous Wi-Fi-based activity recognition methods can realize satisfied recognitions, however, these methods may fail to recognize the activities of an unknown person without the learning process. In this study, using channel state information (CSI) data, a novel cross-person activity recognition (CPAR) method is proposed by a deep learning approach with generalization capability. Combining one of the state-of-the-art deep neural networks (DNNs) used in activity recognition, i.e., attention-based bi-directional long short-term memory (ABLSTM), the snapshot ensemble is the first to be adopted to train several base-classifiers for enhancing the generalization and practicability of recognition. Second, to discriminate the extracted features, metric learning is further introduced by using the center loss, obtaining snapshot ensemble-used ABLSTM with center loss (SE-ABLSTM-C). In the experiments of CPAR, the proposed SE-ABLSTM-C method markedly improved the recognition accuracies to an application level, for seven categories of activities.
BibTeX
@inproceedings{ye2024non,
  author = {Chen Ye and Siyuan Xu and Zhengran He and Yue Yin and Tomoaki Ohtsuki and Guan Gui},
  title = {{Non-Contact Cross-Person Activity Recognition by Deep Metric Ensemble Learning}},
  booktitle = {Bioengineering},
  year = {2024},
  doi = {10.3390/bioengineering11111124},
}
Figure