More Reliable Neighborhood Contrastive Learning for Novel Class Discovery in Sensor-Based Human Activity Recognition

Mingcong Zhang, Tao Zhu, Mingxing Nie, Zhenyu LiuSensors 2023 · DOI

TL;DR

This paper introduces a novel framework, More Reliable Neighborhood Contrastive Learning (MRNCL), for discovering new human activities from unlabeled sensor data. By leveraging existing labeled data and a more robust similarity measure for identifying relevant neighbors, MRNCL significantly improves the accuracy of classifying previously unseen activities in real-world scenarios.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) systems are great at identifying known activities using sensor data. However, a major challenge arises in real-world applications where new activities emerge constantly, and manually labeling all this new data is impractical and expensive. Existing HAR systems struggle to automatically discover and classify these "novel" activities without prior knowledge. This limitation hinders their adaptability and usefulness in dynamic environments. The paper addresses this by framing the problem as "Novel Class Discovery" (NCD), aiming to classify new activities using a model trained on both existing labeled data and unlabeled data containing these new activities.

Methods

The core of the proposed solution is the More Reliable Neighborhood Contrastive Learning (MRNCL) framework. It builds upon the Neighborhood Contrastive Learning (NCL) approach but introduces key improvements:

  • Lightweight Architecture: MRNCL is designed to be more efficient than its predecessor by removing certain components.
  • Novel Similarity Measure: The most significant contribution is a new similarity metric. Traditional NCL uses cosine similarity to find "k-nearest neighbors" (KNNs) in the feature space, which are treated as "pseudo-positives" for contrastive learning. However, cosine similarity can be unreliable. MRNCL introduces a more robust measure that combines Gower, Lorentzian, Dice, and Jaccard distances. This new metric aims to identify KNNs that are truly more likely to belong to the same semantic category as the query sample, leading to more reliable pseudo-positives.
  • Contrastive Learning: The framework employs contrastive learning to learn discriminative features. It pulls the query sample closer to its reliable pseudo-positives and pushes it away from other samples (negatives).
  • Supervised Contrastive Learning (SCL): MRNCL also incorporates SCL on the labeled data to further enhance feature learning.
  • End-to-End Training: The entire framework, including a feature extractor (DeepConvLSTM) and classifiers, is trained end-to-end.

Datasets

The MRNCL framework was evaluated on three publicly available sensor-based HAR datasets:

  • WISDM: Collected using an accelerometer, with 6 activities.
  • UCI-HAR: Collected using accelerometer and gyroscope sensors, with 6 activities.
  • USC-HAD: Collected using accelerometer and gyroscope sensors, with 12 activities.

For the NCD task, these datasets were partitioned into labeled and unlabeled sets. For WISDM and UCI-HAR, 3 activities were used for labeling and the remaining 3 for unlabeled data. For USC-HAD, 6 activities were used for labeling and the other 6 for unlabeled data.

Key Insights and Ideas

  • The Importance of Reliable Neighbors: The paper strongly emphasizes that the quality of the "pseudo-positive" neighbors selected for contrastive learning is paramount. Simply using cosine similarity is insufficient and can lead to misclassifications, especially when dealing with subtle inter-activity similarities.
  • Inter-Activity Similarity is a Major Hurdle: The study highlights that even with advanced methods, activities with high inter-class similarity (e.g., running and jumping) remain challenging to distinguish, impacting recall. This is a fundamental limitation in sensor-based HAR that requires further investigation.
  • Prior Knowledge is Crucial: The experiments demonstrate that leveraging labeled data (even for a subset of activities) significantly boosts the performance of novel class discovery, validating the core premise of the NCD task.
  • Simplicity Can Be Effective: By removing consistency loss and augment-positives (components from the original NCL), MRNCL achieves better or comparable results while being more lightweight, suggesting that a focused approach on reliable neighborhood selection is key.

Key Results

  • Superior Clustering Accuracy: MRNCL consistently outperformed existing methods, including k-means, Agglomerative Clustering, ModifiedNCL, and the original NCL, across all three datasets in terms of clustering accuracy (ACC) and pairwise F-score. For instance, on WISDM, MRNCL achieved an ACC of 83.78% compared to the baseline of 80.97%. On UCI-HAR, it reached 86.25% versus the baseline's 84.53%.
  • Improved Precision and Recall: The method generally showed improvements in both precision and recall, indicating better identification of true positives and fewer false positives/negatives.
  • Robustness: The proposed similarity measure led to more stable training, as evidenced by smoother loss curves compared to methods relying on cosine similarity.
  • Lightweight Design: MRNCL is noted to be more lightweight than NCL, achieving strong results without certain components.

Limitations and Future Work

  • Inter-Activity Similarity: The paper acknowledges that high inter-activity similarity remains a significant challenge, particularly for activities like running and jumping, which can lead to decreased recall. This is identified as a crucial area for future research.
  • Dataset Specificity: While the method performs well across three datasets, the paper notes that the effectiveness of data augmentation (used in contrastive learning) can vary across datasets, suggesting that finding universally effective augmentation strategies for HAR is still an open problem.
  • Future Directions: The authors hope their work will inspire further research into novel class discovery in sensor-based HAR, particularly in addressing the challenges posed by inter-activity similarity.

Summary updated

Abstract

Human Activity Recognition (HAR) systems have made significant progress in recognizing and classifying human activities using sensor data from a variety of sensors. Nevertheless, they have struggled to automatically discover novel activity classes within massive amounts of unlabeled sensor data without external supervision. This restricts their ability to classify new activities of unlabeled sensor data in real-world deployments where fully supervised settings are not applicable. To address this limitation, this paper presents the Novel Class Discovery (NCD) problem, which aims to classify new class activities of unlabeled sensor data by fully utilizing existing activities of labeled data. To address this problem, we propose a new end-to-end framework called More Reliable Neighborhood Contrastive Learning (MRNCL), which is a variant of the Neighborhood Contrastive Learning (NCL) framework commonly used in visual domain. Compared to NCL, our proposed MRNCL framework is more lightweight and introduces an effective similarity measure that can find more reliable <i>k</i>-nearest neighbors of an unlabeled query sample in the embedding space. These neighbors contribute to contrastive learning to facilitate the model. Extensive experiments on three public sensor datasets demonstrate that the proposed model outperforms existing methods in the NCD task in sensor-based HAR, as indicated by the fact that our model performs better in clustering performance of new activity class instances.
BibTeX
@inproceedings{zhang2023more,
  author = {Mingcong Zhang and Tao Zhu and Mingxing Nie and Zhenyu Liu},
  title = {{More Reliable Neighborhood Contrastive Learning for Novel Class Discovery in Sensor-Based Human Activity Recognition}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23239529},
}
Figure