Semantically Encoding Activity Labels for Context-Aware Human Activity Recognition

Wen Ge, Guanyi Mou, Emmanuel O. Agu, Kyumin Lee

Abstract

Traditional Context-Aware Human Activity Recognition (CA-HAR) systems treat activity labels as independent binary targets, ignoring the inherent semantic relationships between actions. This paper introduces SEAL (Semantically Encoding Activity Labels), a framework that leverages Language Models (LMs) to map sensor data into a shared semantic space. By aligning time-series embeddings with textual descriptions, SEAL achieves significant performance improvements across multiple datasets, particularly for rare and nuanced activities. Our results demonstrate improvements of up to 22.6% in Matthews Correlation Coefficient (MCC), highlighting the value of semantic awareness in sensor-based classification.

Introduction

Context-Aware Human Activity Recognition (CA-HAR) is a cornerstone of ubiquitous computing, enabling smartphones and wearables to identify user actions (e.g., walking, sitting) and their environmental context (e.g., phone in pocket, in the bathroom). Traditionally, this task is framed as a multi-label classification problem where each activity is a discrete binary bit.

However, this approach is fundamentally limited because it ignores the semantic proximity between activities. To a standard model, "Walking" and "Jogging" are as mathematically distinct as "Walking" and "Sleeping." In reality, walking and jogging share similar physiological and movement patterns. By treating labels as independent entities, traditional models struggle with noisy sensor data, rare activities, and "confusable" actions that appear similar in raw data streams. The SEAL framework addresses this by reframing HAR as a multi-modal alignment task between sensor signals and natural language.

The SEAL Framework

SEAL moves beyond binary labels by using a pre-trained Language Model to encode the rich semantic relationships between human activities. The framework consists of four primary components:

1. CA-HAR Label Encoder

Instead of one-hot vectors, SEAL utilizes a pre-trained BERT model to generate label embeddings. Simple labels are transformed into descriptive sentences (e.g., "Walking" becomes "The user is walking now"), and the high-dimensional vector embedding is extracted from the BERT classification head. This ensures that semantically related activities are positioned closely together in the embedding space.

2. Sensor Data Encoder

This module transforms raw sensor data—such as accelerometer and gyroscope readings—into a latent vector. A significant advantage of SEAL is that it is backbone-agnostic; it can be integrated with various architectures, including Multi-Layer Perceptrons (MLP), Convolutional Neural Networks (CNN), or Graph Neural Networks (GNN).

3. Modal Alignment

During the training phase, the model employs a custom objective function to align sensor embeddings with their corresponding label embeddings. It uses Cross-Entropy for mutually exclusive contexts and Binary Cross-Entropy for co-occurring activities, forcing the model to learn a shared vector space where sensor patterns match their semantic descriptions.

4. Inference

To recognize an activity, the model performs a similarity search. It calculates which label embedding is closest to the input sensor representation in the shared space, allowing for more nuanced classification than a simple threshold-based binary output.

Experimental Evaluation

The SEAL framework was rigorously evaluated on three real-world datasets representing different levels of environmental control:

  • WASH Scripted: Data collected in controlled laboratory environments with high-fidelity labels.
  • WASH Unscripted: "In the wild" data where participants performed daily activities, resulting in noisier and more diverse signals.
  • Extrasensory: A large-scale public dataset containing sensor data from 60 users in unconstrained, real-world environments.

Key Results and Insights

SEAL consistently outperformed state-of-the-art (SOTA) baselines across all tested datasets. Key findings include:

  • Performance Gains: The framework achieved improvements ranging from 7.8% to 22.6% in Matthews Correlation Coefficient (MCC) and 3.9% to 8.4% in Macro-F1 scores.
  • Semantic Proximity: By utilizing LMs, the model inherently understands that "Running" and "Exercising" are related. This semantic "prior" helps the model maintain accuracy even when sensor signals are subtle or noisy.
  • Rare Activity Recognition: SEAL showed massive improvements (up to 26.3% in MCC) for rare or short-term activities like "Trembling" or "Sneezing." The textual descriptions provided enough guidance for the model to identify these actions even with limited training examples.
  • Contextual Robustness: On unscripted datasets, SEAL significantly improved the recognition of phone placement (e.g., "In Hand" vs. "In Pocket"), a task that is notoriously difficult in unconstrained environments.

Discussion and Future Work

The success of SEAL suggests that the primary bottleneck in HAR may not be the sensor models themselves, but rather the way labels are represented. By moving to a semantic embedding space, SEAL provides a more robust and flexible foundation for activity recognition.

However, challenges remain. The use of BERT adds computational overhead compared to simple classifiers, though smaller LMs could mitigate this. Additionally, the current reliance on manual templates for sentence generation could be improved through automated prompt engineering. Future research will explore expanding SEAL to include other modalities, such as audio or video, to further enrich the semantic alignment and improve recognition in complex, multi-modal environments.

Summary updated

Abstract

Prior work has primarily formulated Context-Aware Human Activity Recognition (CA-HAR) as a multi-label classification problem, where model inputs are time-series sensor data and target labels are binary encodings representing whether a given activity or context occurs. These CA-HAR methods either predicted each label independently or manually imposed relationships using graphs. However, both strategies often neglect an essential aspect: activity labels have rich semantic relationships. For instance, walking, jogging, and running activities share similar movement patterns but differ in pace and intensity, indicating that they are semantically related. Consequently, prior CA-HAR methods often struggled to accurately capture these inherent and nuanced relationships, particularly on datasets with noisy labels typically used for CA-HAR or situations where the ideal sensor type is unavailable (e.g., recognizing speech without audio sensors). To address this limitation, we propose Semantically Encoding Activity Labels (SEAL), which leverage Language Models (LMs) to encode CA-HAR activity labels to capture semantic relationships. LMs generate vector embeddings that preserve rich semantic information from natural language. Our SEAL approach encodes input-time series sensor data from smart devices and their associated activity and context labels (text) as vector embeddings. During training, SEAL aligns the sensor data representations with their corresponding activity/context label embeddings in a shared embedding space. At inference time, SEAL performs a similarity search, returning the CA-HAR label with the embedding representation closest to the input data. Although LMs have been widely explored in other domains, surprisingly, their potential in CA-HAR has been underexplored, making our approach a novel contribution to the field. Our SEAL approach has been rigorously evaluated on three real-world datasets, demonstrating its superior performance. It consistently outperforms state-of-the-art methods by 7.8% to 22.6% in MCC and 3.9% to 8.4% in Macro-F1. Furthermore, SEAL performance is agnostic to the data encoding framework utilized, enhancing performance by 4.7% to 73.3% in MCC and 2.6% to 30.5% in Macro-F1 across different data encoding models. This robust performance opens up new possibilities for integrating more advanced LMs into CA-HAR tasks. We share our code and supplement material at https://github.com/GMouYes/SEAL.
BibTeX
@inproceedings{ge2025semantically,
  author = {Wen Ge and Guanyi Mou and Emmanuel O. Agu and Kyumin Lee},
  title = {{Semantically Encoding Activity Labels for Context-Aware Human Activity Recognition}},
  booktitle = {PerCom},
  year = {2025},
  url = {https://openreview.net/forum?id=1cnXLliN5n},
}
Figure