RobustHAR: Multi-scale Spatial-temporal Masked Self-supervised Pre-training for Robust Human Activity Recognition

Xiao Liu, Guan Yuan, Yanmei Zhang, Shang Liu, Qiuyan YanIJCAI 2025 · DOI

Abstract

RobustHAR is a novel self-supervised framework designed for robust wearable sensor-based human activity recognition (HAR). It addresses the critical challenges of data missingness and spatial-temporal complexity by integrating location-aware 3D spatial modeling with a multi-scale masking strategy across segment, channel, and sensor levels. By enforcing semantic consistency across these scales and utilizing adaptive feature fusion, RobustHAR learns robust representations that significantly outperform state-of-the-art models, particularly in real-world scenarios prone to sensor failure.

Introduction

Wearable Human Activity Recognition (WSHAR) is a cornerstone of modern health monitoring and human-computer interaction. However, deploying these systems in the wild presents two significant hurdles. First, data missingness is common due to sensor malfunctions, connection losses, or noise, which often causes traditional supervised models to fail. Second, human movement is characterized by spatial-temporal complexity; it is a coordinated effort across various body parts over varying durations. Existing self-supervised methods frequently prioritize the temporal dimension while neglecting the physical spatial relationships between sensors.

RobustHAR bridges this gap by explicitly modeling the 3D physical locations of sensors and employing a multi-scale masking strategy. This approach allows the model to move beyond simple time-series analysis toward a more "biological" understanding of movement, enabling it to "fill in the blanks" when data is missing.

The RobustHAR Framework

The RobustHAR architecture is built upon a Transformer-based backbone and introduces several innovative components to enhance robustness and feature learning.

The overall architecture of RobustHAR, including (a) spatial-temporal 3D-variation modeling, (b) self-supervised pre-training, (c) fine-tuning, and (d) activity recognition.

Core Components:

  • Location-inspired 3D-variation Modeling: Rather than treating sensor data as a flat matrix, RobustHAR maps sensors to a 3D coordinate system based on their physical placement on the body. This captures essential spatial correlations, such as the relationship between wrist and ankle movements.
  • Semantic-consistent Co-learning: To ensure a unified representation of activities, the framework implements a consistency loss. This forces features learned from different masking scales to align, ensuring the model recognizes an activity regardless of whether it observes a small time slice or a specific subset of sensors.
  • Adaptive Multi-scale Feature Fusion: During the fine-tuning phase, a cross-attention mechanism intelligently weighs and combines features from different scales to produce the final classification.

Multi-scale Spatial-temporal Masked Pre-training

A key innovation of RobustHAR is its pre-training strategy, which uses masking as a proxy for real-world sensor failure. By learning to reconstruct hidden data, the model becomes inherently resilient to hardware malfunctions.

Illustration of the three masking scales used in RobustHAR: segment-wise, channel-wise, and sensor-wise masking.

As shown in the figure above, the model is trained to reconstruct data masked at three distinct levels:

  1. Segment-wise Masking: Hiding blocks of time to capture temporal dependencies.
  2. Channel-wise Masking: Hiding specific sensor axes (e.g., the X-axis of an accelerometer) to learn intra-sensor correlations.
  3. Sensor-wise Masking: Hiding the entire data stream from a physical device to learn inter-sensor spatial dependencies.

Experimental Evaluation

The model was evaluated on three prominent multi-sensor public datasets: Opportunity (5 devices), RealWorld (7 devices), and CZU-MHAD (10 devices). To simulate "low-label" real-world scenarios, only 15% of the labeled data was used for fine-tuning.

Key Results:

  • State-of-the-Art Performance: RobustHAR consistently outperformed supervised baselines (e.g., Informer, TimeNet) and other self-supervised methods (e.g., SimMTM, ST-MAE), achieving accuracies of 93.72% on Opportunity, 95.76% on RealWorld, and 94.63% on CZU-MHAD.
  • Resilience to Missing Data: In stress tests where 15% of the data was randomly removed, RobustHAR maintained an accuracy of 86-89%, demonstrating significantly less performance degradation than competing models.
  • Ablation Studies: Removing the multi-scale masking module resulted in a 10% accuracy drop on the RealWorld dataset, confirming that multi-scale integration is vital for the model's success.

Conclusion and Future Work

RobustHAR demonstrates that integrating spatial awareness with multi-scale self-supervision creates a highly resilient HAR system. While the model's computational complexity is slightly higher than simpler alternatives, its robustness to missing data makes it a strong candidate for real-world applications. Future work will focus on optimizing the model for real-time inference on low-power edge devices and extending the framework to handle heterogeneous sensor types, such as combining IMUs with heart rate or EMG sensors.

Summary updated

BibTeX
@inproceedings{liu2025robusthar,
  author = {Xiao Liu and Guan Yuan and Yanmei Zhang and Shang Liu and Qiuyan Yan},
  title = {{RobustHAR: Multi-scale Spatial-temporal Masked Self-supervised Pre-training for Robust Human Activity Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/952},
}
Figure