Heterogeneous Hyper-Graph Neural Networks for Context-Aware Human Activity Recognition

Wen Ge, Guanyi Mou, Emmanuel O. Agu, Kyumin LeePerCom Workshops 2023 · DOI · OpenReview

Abstract

Context-aware Human Activity Recognition (CHAR) is challenging due to the need to recognize the user's current activity from signals that vary significantly with contextual factors such as phone placements and the varied styles with which different users perform the same activity. In this paper, we argue that context-aware activity visit patterns in realistic in-the-wild data can equivocally be considered as a general graph representation learning task. We demonstrate that CHAR data has an underlying graph structure that can be viewed as a heterogeneous hypergraph that has multiple types of nodes and hyperedges. We propose a novel Heterogeneous HyperGraph Neural Network architecture for Context-aware Human Activity Recognition (HHGNN-CHAR), with three types of heterogeneous nodes (user, phone placement, and activity). Rigorous evaluation demonstrated that on an unscripted, in-the-wild CHAR dataset, our proposed framework significantly outperforms state-of-the-art (SOTA) baselines with overall improvements of 14.04% on Matthews Correlation Coefficient (MCC) and 7.01% on Macro F1 scores.

Introduction

Standard Human Activity Recognition (HAR) identifies activities like walking or sitting. However, Context-aware HAR (CHAR) aims to identify the activity alongside its context—specifically, the phone's location (e.g., <Walking, In Pocket>). This task is notoriously difficult due to signal variability (the same activity produces different sensor data depending on phone placement) and user diversity (individual movement styles).

Existing models often treat these as independent classification tasks, failing to exploit the inherent relational nature of the data. For instance, certain users frequently perform specific activities with their phones in specific positions. This research transforms CHAR into a graph representation learning problem to capture these complex dependencies.

Methodology

Heterogeneous Hypergraph Representation

The authors propose modeling CHAR data as a Heterogeneous Hypergraph. Unlike standard graphs where edges connect two nodes, a hyperedge can connect multiple nodes simultaneously. In this framework:

  • Nodes represent three distinct entities: Users ($U$), Phone Placements ($PP$), and Activities ($ACT$).
  • Hyperedges represent specific instances where a user performs an activity with a specific phone placement.

The HHGNN-CHAR Architecture

The proposed HHGNN-CHAR model processes this graph through three specialized layers designed to handle the complexity of heterogeneous data and multi-way relationships.

Architecture of the HHGNN-CHAR framework, illustrating the Heterogeneity, Hypergraph, and Context-aware HAR layers.
  1. Heterogeneity Layer: This layer accounts for the fundamental differences between node types (e.g., a "User" node vs. an "Activity" node) by projecting them into distinct mathematical spaces using type-specific linear parameters.
  2. Hypergraph Layer: Utilizing Hypergraph Convolutions (HyperConv), this layer allows information to flow across hyperedges, capturing the "neighborhood" context of how users, activities, and placements interact.
  3. Context-aware HAR Layer: This final layer takes new sensor data, treats it as a candidate hyperedge, and calculates similarity against learned node embeddings to predict the most likely Activity and Placement tuple.

Experimental Evaluation

The model was evaluated on the Extrasensory dataset, a benchmark containing over 6.3 million instances from 60 participants. This "in-the-wild" dataset includes 170 features (accelerometer, gyroscope, audio, etc.) and labels for 4 phone placements and 13 activities. Its unscripted nature provides a realistic test for the model's ability to handle messy, overlapping daily life data.

Results and Discussion

HHGNN-CHAR significantly outperformed strong baselines, including LightGBM and standard Graph Convolutional Networks (GCNs). The inclusion of the "User" as a node allowed the model to implicitly learn personal movement styles without requiring explicit user metadata.

Performance comparison of HHGNN-CHAR against state-of-the-art baselines on the Extrasensory dataset.

As shown in the results, HHGNN-CHAR achieved an 84.5% MCC and 91.6% Macro F1 score, outperforming the best non-graph baseline (LightGBM) by 14.04% in MCC. Ablation studies further confirmed that both the heterogeneity component and the hypergraph structure are critical; removing either led to significant performance degradation.

Conclusion

By treating CHAR as a relational graph problem, HHGNN-CHAR effectively captures the multi-way interactions between users, activities, and contexts. Future work will focus on user generalization (testing on entirely new subjects) and exploring alternative GNN backbones to further optimize performance and computational efficiency.

Summary updated

Abstract

Context-aware Human Activity Recognition (CHAR) is challenging due to the need to recognize the user's current activity from signals that vary significantly with contextual factors such as phone placements and the varied styles with which different users perform the same activity. In this paper, we argue that context-aware activity visit patterns in realistic in-the-wild data can equivocally be considered as a general graph representation learning task. We posit that exploiting underlying graphical patterns in CHAR data can improve CHAR task performance and representation learning. Building on the intuition that certain activities are frequently performed with the phone placed in certain positions, we focus on the context-aware human activity problem of recognizing the <Activity, Phone Placement> tuple. We demonstrate that CHAR data has an underlying graph structure that can be viewed as a heterogenous hypergraph that has multiple types of nodes and hyperedges (an edge connecting more than two nodes). Subsequently, learning <Activity, Phone Placement> representations becomes a graph node representation learning problem. After task transformation, we further propose a novel Heterogeneous HyperGraph Neural Network architecture for Context-aware Human Activity Recognition (HHGNN-CHAR), with three types of heterogeneous nodes (user, phone placement, and activity). Connections between all types of nodes are represented by hyperedges. Rigorous evaluation demonstrated that on an unscripted, in-the-wild CHAR dataset, our proposed framework significantly outperforms state-of-the-art (SOTA) baselines including CHAR models that do not exploit graphs, and GNN variants that do not incorporate heterogeneous nodes or hyperedges with overall improvements 14.04% on Matthews Correlation Coefficient (MCC) and 7.01% on Macro F1 scores.
BibTeX
@inproceedings{ge2023heterogeneous,
  author = {Wen Ge and Guanyi Mou and Emmanuel O. Agu and Kyumin Lee},
  title = {{Heterogeneous Hyper-Graph Neural Networks for Context-Aware Human Activity Recognition}},
  booktitle = {PerCom Workshops},
  year = {2023},
  doi = {10.1109/percomworkshops56833.2023.10150328},
}
Figure