Abstract
With the popularity and development of wearable devices such as smartphones, human activity recognition (HAR) based on sensors has become a key research area in human-computer interaction and ubiquitous computing. The emergence of deep learning has led to a recent shift in HAR research, which traditionally requires massive strictly labeled data. In comparison with video data, activity data recorded from an accelerometer or gyroscope is often more difficult to interpret and segment. Recently, several attention mechanisms have been proposed to handle weakly labeled human activity data, which do not require accurate data annotation. However, these models are often limited to segments containing a single labeled activity. In this paper, we propose a recurrent attention network (RAN) to handle sequential activity recognition and location tasks. The model repeatedly performs steps of attention on multiple activities within one segment, with each step corresponding to a focused activity based on previous observations. Experimental results on the UniMiB-SHAR dataset and a newly collected sequential weakly labeled multi-activity dataset show that our model can effectively recognize and locate sequential activities, facilitating the process of sensor data accumulation through automatic segmentation.
Introduction: The Challenge of Weakly Labeled Data
Deep learning has revolutionized Human Activity Recognition (HAR) using wearable sensors. However, these models typically require "strictly labeled" datasets, where human annotators must precisely mark the start and end times of every activity. For noisy sensor data like that from accelerometers, this process is tedious and error-prone.
While "weakly labeled" learning exists—where only the presence of an activity in a segment is noted—most current models can only handle segments containing a single activity. Real-world data is inherently sequential; movements transition naturally from one to another (e.g., walking to sitting). This research addresses the Sequential Weakly Labeled Multi-Activity (SWLM) task: identifying multiple different activities in a long sequence and pinpointing their locations without precise ground-truth timestamps during training.
Proposed Architecture: Recurrent Attention Network (RAN)
The proposed RAN architecture functions through a "glimpse" mechanism, allowing the model to look at a long sequence multiple times to identify different activities. It consists of three primary components:
- CNN Feature Extractor: A 1D-Convolutional Neural Network acts as an encoder, transforming raw, noisy sensor signals into high-level feature vectors.
- Attention Module: This component calculates a weight for every time step in the sequence, identifying which parts of the signal are most relevant to the current activity being classified.
- LSTM Decoder: The Long Short-Term Memory (LSTM) network acts as the controller. It maintains a hidden state that tracks previously identified activities. At each step $t$, it uses its memory to direct the Attention Module where to look next and then classifies the focused activity.
By using an LSTM, the model learns the transitions between activities. For example, if it has just identified "Sitting," its internal state helps the attention mechanism find the likely next movement, such as "Standing."
Key Results and Insights
The model was evaluated on standard benchmarks and a newly collected SWLM dataset. Key findings include:
- Competitive Performance: On the UniMiB-SHAR dataset for single activity recognition, the RAN achieved 72.8% accuracy, slightly outperforming a standard CNN baseline.
- Multi-Activity Mastery: On the new SWLM dataset, the model achieved 99% accuracy in identifying activities in known sequences. Even in "reverse order" tests (training on A-B, testing on B-A), it maintained 85.6% accuracy, proving it recognizes underlying patterns rather than just memorizing sequences.
- Automated Localization: The attention weights effectively serve as timestamps. By smoothing these weights, the model can automatically generate bounding boxes for activities, significantly reducing the manual labor required for data annotation.
Conclusion
The Recurrent Attention Network provides a robust solution for recognizing and locating multiple activities in sequential, weakly labeled sensor data. By automating the segmentation process, this approach lowers the barrier to creating large-scale, high-quality datasets for HAR, paving the way for more practical and efficient wearable computing applications.
Summary updated