Abstract
Traditional Context-Aware Human Activity Recognition (CA-HAR) systems treat activity labels as independent binary targets, ignoring the inherent semantic relationships between actions. This paper introduces SEAL (Semantically Encoding Activity Labels), a framework that leverages Language Models (LMs) to map sensor data into a shared semantic space. By aligning time-series embeddings with textual descriptions, SEAL achieves significant performance improvements across multiple datasets, particularly for rare and nuanced activities. Our results demonstrate improvements of up to 22.6% in Matthews Correlation Coefficient (MCC), highlighting the value of semantic awareness in sensor-based classification.
Introduction
Context-Aware Human Activity Recognition (CA-HAR) is a cornerstone of ubiquitous computing, enabling smartphones and wearables to identify user actions (e.g., walking, sitting) and their environmental context (e.g., phone in pocket, in the bathroom). Traditionally, this task is framed as a multi-label classification problem where each activity is a discrete binary bit.
However, this approach is fundamentally limited because it ignores the semantic proximity between activities. To a standard model, "Walking" and "Jogging" are as mathematically distinct as "Walking" and "Sleeping." In reality, walking and jogging share similar physiological and movement patterns. By treating labels as independent entities, traditional models struggle with noisy sensor data, rare activities, and "confusable" actions that appear similar in raw data streams. The SEAL framework addresses this by reframing HAR as a multi-modal alignment task between sensor signals and natural language.
The SEAL Framework
SEAL moves beyond binary labels by using a pre-trained Language Model to encode the rich semantic relationships between human activities. The framework consists of four primary components:
1. CA-HAR Label Encoder
Instead of one-hot vectors, SEAL utilizes a pre-trained BERT model to generate label embeddings. Simple labels are transformed into descriptive sentences (e.g., "Walking" becomes "The user is walking now"), and the high-dimensional vector embedding is extracted from the BERT classification head. This ensures that semantically related activities are positioned closely together in the embedding space.
2. Sensor Data Encoder
This module transforms raw sensor data—such as accelerometer and gyroscope readings—into a latent vector. A significant advantage of SEAL is that it is backbone-agnostic; it can be integrated with various architectures, including Multi-Layer Perceptrons (MLP), Convolutional Neural Networks (CNN), or Graph Neural Networks (GNN).
3. Modal Alignment
During the training phase, the model employs a custom objective function to align sensor embeddings with their corresponding label embeddings. It uses Cross-Entropy for mutually exclusive contexts and Binary Cross-Entropy for co-occurring activities, forcing the model to learn a shared vector space where sensor patterns match their semantic descriptions.
4. Inference
To recognize an activity, the model performs a similarity search. It calculates which label embedding is closest to the input sensor representation in the shared space, allowing for more nuanced classification than a simple threshold-based binary output.
Experimental Evaluation
The SEAL framework was rigorously evaluated on three real-world datasets representing different levels of environmental control:
- WASH Scripted: Data collected in controlled laboratory environments with high-fidelity labels.
- WASH Unscripted: "In the wild" data where participants performed daily activities, resulting in noisier and more diverse signals.
- Extrasensory: A large-scale public dataset containing sensor data from 60 users in unconstrained, real-world environments.
Key Results and Insights
SEAL consistently outperformed state-of-the-art (SOTA) baselines across all tested datasets. Key findings include:
- Performance Gains: The framework achieved improvements ranging from 7.8% to 22.6% in Matthews Correlation Coefficient (MCC) and 3.9% to 8.4% in Macro-F1 scores.
- Semantic Proximity: By utilizing LMs, the model inherently understands that "Running" and "Exercising" are related. This semantic "prior" helps the model maintain accuracy even when sensor signals are subtle or noisy.
- Rare Activity Recognition: SEAL showed massive improvements (up to 26.3% in MCC) for rare or short-term activities like "Trembling" or "Sneezing." The textual descriptions provided enough guidance for the model to identify these actions even with limited training examples.
- Contextual Robustness: On unscripted datasets, SEAL significantly improved the recognition of phone placement (e.g., "In Hand" vs. "In Pocket"), a task that is notoriously difficult in unconstrained environments.
Discussion and Future Work
The success of SEAL suggests that the primary bottleneck in HAR may not be the sensor models themselves, but rather the way labels are represented. By moving to a semantic embedding space, SEAL provides a more robust and flexible foundation for activity recognition.
However, challenges remain. The use of BERT adds computational overhead compared to simple classifiers, though smaller LMs could mitigate this. Additionally, the current reliance on manual templates for sentence generation could be improved through automated prompt engineering. Future research will explore expanding SEAL to include other modalities, such as audio or video, to further enrich the semantic alignment and improve recognition in complex, multi-modal environments.
Summary updated