Domain-Informed Label Fusion Surpasses LLMs in Free-Living Activity Classification (Student Abstract)

Shovito Barua Soumma, Abdullah Mamun, Hassan GhasemzadehAAAI 2025 · DOI

Introduction

Human Activity Recognition (HAR) has achieved significant success in controlled laboratory settings. However, transitioning these models to "free-living" environments—the real world—presents a major challenge: the "noisy label" problem. When participants self-report activities, they use inconsistent, open-ended text. For example, "walking," "strolling," and "going to the shop" might all describe the same physical state but appear as distinct labels in a dataset. This results in a sparse, high-cardinality vocabulary that standard machine learning models struggle to generalize, especially under the constraints of low-frequency sensor sampling common in wearable devices.

While Large Language Models (LLMs) like GPT-4 are often employed to clean and standardize text, they frequently lack the domain-specific physiological understanding required for HAR. To address this, we present FuSE-MET (Fusion of Semantic Embeddings and MET Values), a framework that cleans noisy labels by integrating linguistic meaning with physical intensity data.

The FuSE-MET Framework

The core innovation of FuSE-MET is the mapping of user-provided labels into a multi-dimensional feature vector that captures both semantic and physiological characteristics.

The FuSE-MET framework for merging noisy activity labels using semantic embeddings and physical intensity (MET) values.

The framework utilizes a weighted fusion formula to represent each activity $a_i$: $f(a_i) = (1 - \lambda) \cdot w(a_i) + \lambda \cdot m(a_i)$

Here, $w(a_i)$ represents BERT-based word embeddings that capture linguistic relationships (e.g., the proximity of "shopping" to "buying"). The term $m(a_i)$ represents Metabolic Equivalent of Task (MET) values, a clinical standard for measuring energy expenditure. The hyperparameter $\lambda$ balances these two inputs. Following fusion, labels are grouped via K-means clustering to create a refined, manageable label space for training classifiers like XGBoost or SVM on smartphone accelerometer and gyroscope data.

Experimental Evaluation

The framework was evaluated using data from a clinical study of patients with cardiovascular disease. Sensor data was segmented into 5-second windows with a 50% duty cycle to simulate battery-saving constraints. The original dataset contained over 36 unique, noisy labels provided by users through active learning.

Key Results

FuSE-MET consistently outperformed both the "no-fusion" baseline and GPT-4-based label merging across various cluster sizes ($K$).

Performance comparison of FuSE-MET against GPT-4-based merging and a no-fusion baseline across different cluster sizes (K).

As shown in the results, the best performance was achieved at $K=3$ using an XGBoost classifier, yielding an F1-score of 0.77—significantly higher than the 0.51 achieved by a GPT-4-based baseline. At a coarser granularity ($K=2$), the model achieved a near-perfect accuracy of 0.98 using an SVM. In contrast, attempting to classify the original 36 noisy labels without any fusion resulted in a dismal F1-score of 0.18, underscoring the necessity of label cleaning in free-living HAR.

Discussion and Insights

The superiority of FuSE-MET over LLMs suggests that domain knowledge acts as a vital regularizer. While GPT-4 is adept at language, it lacks the "physical intuition" to distinguish between activities that are semantically similar but physically distinct. Integrating MET values prevents the model from incorrectly merging low-intensity and high-intensity activities.

Furthermore, the optimal $\lambda$ values (typically between 0.2 and 0.5) indicate that while physical intensity is a critical anchor, semantic meaning remains the primary driver for successful activity classification. By reducing label complexity, FuSE-MET allows classical classifiers to achieve high performance without overfitting to the noise inherent in raw user input.

Conclusion and Future Work

FuSE-MET demonstrates that domain-informed label fusion is a powerful tool for real-world HAR. Future research will focus on optimizing the mapping of MET values into the embedding space, testing the framework on more diverse populations, and developing methods to automatically learn the optimal balance between semantic and physical features.

Summary updated

BibTeX
@inproceedings{soumma2025domain,
  author = {Shovito Barua Soumma and Abdullah Mamun and Hassan Ghasemzadeh},
  title = {{Domain-Informed Label Fusion Surpasses LLMs in Free-Living Activity Classification (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35301},
}
Figure