Unsupervised Human Activity Recognition through Two-stage Prompting with ChatGPT

Qingxin Xia, Takuya Maekawa, Takahiro HaraarXiv 2023 · DOI · arXiv

TL;DR

This paper introduces a novel unsupervised approach to Human Activity Recognition (HAR) by leveraging ChatGPT's language understanding capabilities. Instead of relying on manually crafted activity descriptions, the method uses a two-stage prompting strategy to automatically generate contextually rich descriptions from sequences of object usage, achieving state-of-the-art performance on benchmark datasets.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) is crucial for various applications, from personal health monitoring to smart environments. However, most HAR methods require extensive labeled data for training, which is costly and time-consuming to acquire. Unsupervised HAR methods aim to overcome this limitation, but existing approaches often depend on manually prepared descriptions of activities, limiting their scalability and performance. This paper addresses the challenge of performing HAR in an unsupervised manner by effectively utilizing the vast knowledge embedded within large language models like ChatGPT, specifically focusing on how object usage sequences can be interpreted to identify activities without explicit human supervision.

Methods

The core innovation lies in a "two-stage prompt engineering" approach designed to guide ChatGPT.

  • Stage 1: Knowledge Generation: This stage focuses on identifying pairs of activities that are difficult to distinguish based solely on object usage. The first prompt asks ChatGPT to list such ambiguous activity pairs. The second prompt then leverages these identified pairs to generate detailed descriptions of each activity, emphasizing the objects that are crucial for differentiating them. This process aims to create a rich, context-aware knowledge base for HAR.
  • Stage 2: Answer Generation: In this stage, the generated knowledge from Stage 1 is used as context for ChatGPT. Given a sequence of objects a user interacted with, a prompt is constructed that includes all possible activity classes and the newly generated activity descriptions. ChatGPT is then asked to predict the most likely activity and provide an explanation. This stage also incorporates "knowledge prompt engineering" by asking for an explanation, which encourages ChatGPT to focus on the important object-based contexts for accurate classification.

Datasets

The proposed method was evaluated on three publicly available datasets:

  • Opportunity dataset: A widely used benchmark for HAR from wearables, featuring five activity classes. Object labels are recorded when users interact with objects.
  • 50 Salads dataset: Collects accelerometer data from a kitchen environment, with three activity classes related to cooking. Object labels are captured during object usage.
  • CMU-MMAC dataset: A multi-modal database for cooking and food preparation tasks, involving twenty-five subjects and twenty-five object types.

Each dataset provides pre-segmented time periods corresponding to activities, with recorded object usage.

Key Insights and Ideas

  • ChatGPT as a Knowledge Engine for HAR: The paper demonstrates that LLMs like ChatGPT, when properly prompted, can act as powerful knowledge generators for unsupervised HAR, bridging the gap between raw sensor data (object sequences) and meaningful activity labels.
  • The Power of Differentiating Similar Activities: The two-stage approach, by first identifying and then describing ambiguous activity pairs, is key. This forces ChatGPT to focus on the subtle differences in object usage that distinguish similar activities, leading to more robust and discriminative descriptions.
  • Unsupervised HAR with Object Sequences is Feasible: The work validates the hypothesis that sequences of object usage, when interpreted by a sophisticated language model, can be sufficient for unsupervised HAR, even in datasets with significant object overlap between activities.
  • Prompt Engineering is Crucial for LLM-based HAR: The success hinges on carefully designed prompts that guide ChatGPT to extract and utilize relevant information, moving beyond simple zero-shot or retrieval-based methods.

Key Results

The proposed two-stage prompting approach achieved state-of-the-art performance across all three evaluated datasets, outperforming baseline methods like zero-shot prompting, retrieval-based knowledge prompting, and few-shot prompting.

  • Opportunity Dataset: Achieved an F1-score of 91.15%, significantly higher than zero-shot (53.61%), retrieval-based (48.19%), and few-shot (73.08%).
  • 50 Salads Dataset: Achieved a perfect F1-score of 100.00%, outperforming baselines (64.47%, 93.87%, 90.61%).
  • CMU-MMAC Dataset: Also achieved a perfect F1-score of 100.00%, outperforming baselines (76.32%, 71.05%, 100.00% - though few-shot also reached 100%, the proposed method is more unsupervised).

The confusion matrix for the Opportunity dataset also showed that the proposed method successfully distinguished the "cleanup" activity, which had significant object overlap with other activities.

Limitations and Future Work

  • Reliance on Pre-trained LLM: The performance is dependent on the knowledge embedded within the pre-trained ChatGPT model. Fine-tuning the LLM for specific HAR tasks could potentially yield even better results.
  • Computational Cost: While unsupervised, the use of LLMs can be computationally intensive.
  • Generalizability to Novel Objects/Activities: The paper doesn't explicitly discuss how the model would handle entirely new objects or activities not present in the training data or the LLM's pre-training.

Future work could explore fine-tuning LLMs for HAR, investigating more efficient prompting strategies, and extending the approach to handle more complex, multi-modal sensor data.

Summary updated

Abstract

Wearable sensor devices, which offer the advantage of recording daily objects used by a person while performing an activity, enable the feasibility of unsupervised Human Activity Recognition (HAR). Unfortunately, previous unsupervised approaches using the usage sequence of objects usually require a proper description of activities manually prepared by humans. Instead, we leverage the knowledge embedded in a Large Language Model (LLM) of ChatGPT. Because the sequence of objects robustly characterizes the activity identity, it is possible that ChatGPT already learned the association between activities and objects from existing contexts. However, previous prompt engineering for ChatGPT exhibits limited generalization ability when dealing with a list of words (i.e., sequence of objects) due to the similar weighting assigned to each word in the list. In this study, we propose a two-stage prompt engineering, which first guides ChatGPT to generate activity descriptions associated with objects while emphasizing important objects for distinguishing similar activities; then outputs activity classes and explanations for enhancing the contexts that are helpful for HAR. To the best of our knowledge, this is the first study that utilizes ChatGPT to recognize activities using objects in an unsupervised manner. We conducted our approach on three datasets and demonstrated the state-of-the-art performance.
BibTeX
@misc{xia2023unsupervised,
  author = {Qingxin Xia and Takuya Maekawa and Takahiro Hara},
  title = {{Unsupervised Human Activity Recognition through Two-stage Prompting with ChatGPT}},
  year = {2023},
  eprint = {2306.02140},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.02140},
}
Figure