Abstract
Cross-modal contrastive pre-training between natural language and other modalities, e.g., vision and audio, has demonstrated astonishing performance and effectiveness across a diverse variety of tasks and domains. In this paper, we investigate whether such natural language supervision can be used for wearable sensor based Human Activity Recognition (HAR), and discover that—surprisingly—it performs substantially worse than standard end-to-end training and self-supervision. We identify the primary causes for this as: sensor heterogeneity and the lack of rich, diverse text descriptions of activities. To mitigate their impact, we also develop strategies and assess their effectiveness through an extensive experimental evaluation. These strategies lead to significant increases in activity recognition, bringing performance closer to supervised and self-supervised training, while also enabling the recognition of unseen activities and cross-modal retrieval of videos. Overall, our work paves the way for better sensor-language learning, ultimately leading to the development of foundational models for HAR using wearables.
Introduction
In fields like computer vision and audio processing, models trained to link images or sounds with natural language descriptions (Natural Language Supervision, or NLS) have enabled powerful "zero-shot" capabilities. This allows models to recognize novel classes simply through text descriptions. This research investigates whether this "plug-and-play" capability translates to Human Activity Recognition (HAR) using wearable sensors, such as accelerometers. Ideally, a user could query a model to "Find data where the user is doing a bicep curl" without specific training on that class. However, initial experiments reveal that standard NLS approaches in HAR perform significantly worse than traditional supervised or self-supervised methods.
Methodology
The researchers implemented a cross-modal contrastive pre-training framework, analogous to CLIP, consisting of two main stages: pre-training and recognition.
Architecture and Loss
The architecture utilizes a convolutional encoder for Inertial Measurement Unit (IMU) data and a transformer-based encoder (e.g., DistilBERT or CLIP-Text) for activity descriptions. A symmetric cross-entropy loss pulls matching sensor-text pairs together in a shared embedding space while pushing mismatched pairs apart.
Mitigation Strategies
To bridge the performance gap, two key strategies were introduced:
- Projection Layer Adaptation: Instead of pure zero-shot inference, the final "projection heads" are updated using a minimal amount of labeled data from the target domain.
- LLM-Augmented Descriptions: To address the sparsity of simple labels (e.g., "walking"), ChatGPT was used to generate 33 diverse templates. Furthermore, "external knowledge" regarding the body parts involved in specific movements was incorporated to enrich text embeddings.
Experimental Setup
Models were pre-trained on the Capture-24 dataset (151 users, 177 activity labels) and evaluated across six diverse target datasets: HHAR, Myogym, Mobiact, Motionsense, MHEALTH, and PAMAP2. These datasets represent various sensor placements (wrist, waist, leg) and activity types.
Results and Key Insights
The experimental evaluation highlighted several critical challenges and successes.
The Performance Gap
Initial zero-shot NLS performance was found to be drastically lower than baselines, often trailing by 30-40% in F1-score. For instance, on the Myogym dataset, NLS achieved only 1.47% F1-score compared to 38.21% for a standard classifier.
Identifying the Causes
- Sensor Heterogeneity: Unlike standardized camera pixels, wearable sensors vary significantly in gain, sampling rates, and hardware noise. This distribution shift hinders "out-of-the-box" generalization across different devices.
- Linguistic Sparsity: HAR labels typically lack the descriptive richness found in vision datasets (e.g., "walking" vs. "a dog playing in a park"). This makes it difficult for models to learn complex activity concepts.
Success of Adaptation
Adapting the model with as few as 100 windows (less than 4 minutes of data) per activity significantly improved performance. On the HHAR dataset, F1-scores rose from ~31% to over 58%, making the model competitive with fully supervised counterparts. Additionally, the inclusion of external knowledge via LLMs boosted the recognition of unseen activities.
Conclusion
While Natural Language Supervision for HAR currently faces hurdles due to sensor heterogeneity and poor text descriptions, this work demonstrates that these can be mitigated. Through projection layer adaptation and LLM-based text augmentation, NLS can approach the performance of supervised models while enabling zero-shot capabilities and cross-modal retrieval. This research paves the way for developing foundational models for human movement.
Summary updated