Sequential Weakly Labeled Multi-Activity Recognition and Location on Wearable Sensors using Recurrent Attention Network

Kun Wang, Jun He, Lei Zhang

Abstract

With the popularity and development of wearable devices such as smartphones, human activity recognition (HAR) based on sensors has become a key research area in human-computer interaction and ubiquitous computing. The emergence of deep learning has led to a recent shift in HAR research, which traditionally requires massive strictly labeled data. In comparison with video data, activity data recorded from an accelerometer or gyroscope is often more difficult to interpret and segment. Recently, several attention mechanisms have been proposed to handle weakly labeled human activity data, which do not require accurate data annotation. However, these models are often limited to segments containing a single labeled activity. In this paper, we propose a recurrent attention network (RAN) to handle sequential activity recognition and location tasks. The model repeatedly performs steps of attention on multiple activities within one segment, with each step corresponding to a focused activity based on previous observations. Experimental results on the UniMiB-SHAR dataset and a newly collected sequential weakly labeled multi-activity dataset show that our model can effectively recognize and locate sequential activities, facilitating the process of sensor data accumulation through automatic segmentation.

Introduction: The Challenge of Weakly Labeled Data

Deep learning has revolutionized Human Activity Recognition (HAR) using wearable sensors. However, these models typically require "strictly labeled" datasets, where human annotators must precisely mark the start and end times of every activity. For noisy sensor data like that from accelerometers, this process is tedious and error-prone.

Illustration of sequential sensor data and activity segments

While "weakly labeled" learning exists—where only the presence of an activity in a segment is noted—most current models can only handle segments containing a single activity. Real-world data is inherently sequential; movements transition naturally from one to another (e.g., walking to sitting). This research addresses the Sequential Weakly Labeled Multi-Activity (SWLM) task: identifying multiple different activities in a long sequence and pinpointing their locations without precise ground-truth timestamps during training.

Proposed Architecture: Recurrent Attention Network (RAN)

The proposed RAN architecture functions through a "glimpse" mechanism, allowing the model to look at a long sequence multiple times to identify different activities. It consists of three primary components:

  1. CNN Feature Extractor: A 1D-Convolutional Neural Network acts as an encoder, transforming raw, noisy sensor signals into high-level feature vectors.
  2. Attention Module: This component calculates a weight for every time step in the sequence, identifying which parts of the signal are most relevant to the current activity being classified.
  3. LSTM Decoder: The Long Short-Term Memory (LSTM) network acts as the controller. It maintains a hidden state that tracks previously identified activities. At each step $t$, it uses its memory to direct the Attention Module where to look next and then classifies the focused activity.
Detailed block diagram of the Recurrent Attention Network (RAN) architecture

By using an LSTM, the model learns the transitions between activities. For example, if it has just identified "Sitting," its internal state helps the attention mechanism find the likely next movement, such as "Standing."

Key Results and Insights

The model was evaluated on standard benchmarks and a newly collected SWLM dataset. Key findings include:

  • Competitive Performance: On the UniMiB-SHAR dataset for single activity recognition, the RAN achieved 72.8% accuracy, slightly outperforming a standard CNN baseline.
  • Multi-Activity Mastery: On the new SWLM dataset, the model achieved 99% accuracy in identifying activities in known sequences. Even in "reverse order" tests (training on A-B, testing on B-A), it maintained 85.6% accuracy, proving it recognizes underlying patterns rather than just memorizing sequences.
  • Automated Localization: The attention weights effectively serve as timestamps. By smoothing these weights, the model can automatically generate bounding boxes for activities, significantly reducing the manual labor required for data annotation.

Conclusion

The Recurrent Attention Network provides a robust solution for recognizing and locating multiple activities in sequential, weakly labeled sensor data. By automating the segmentation process, this approach lowers the barrier to creating large-scale, high-quality datasets for HAR, paving the way for more practical and efficient wearable computing applications.

Summary updated

Abstract

With the popularity and development of the wearable devices such as smartphones, human activity recognition (HAR) based on sensors has become as a key research area in human computer interaction and ubiquitous computing. The emergence of deep learning leads to a recent shift in the research of HAR, which requires massive strictly labeled data. In comparison with video data, activity data recorded from an accelerometer or gyroscope is often more difficult to interpret and segment. Recently, several attention mechanisms are proposed to handle the weakly labeled human activity data, which do not require accurate data annotation. However, these attention-based models can only handle the weakly labeled dataset whose segment includes one labeled activity, as a result it limits efficiency and practicality. In the paper, we proposed a recurrent attention network to handle sequential activity recognition and location tasks. The model can repeatedly perform steps of attention on multiple activities of one segment and each step is corresponding to the current focused activity according to its previous observations. The effectiveness of the recurrent attention model is validated by comparing with a baseline CNN, on the UniMiB-SHAR dataset and a collected sequential weakly labeled multi-activity dataset. The experiment results show that our recurrent attention model not only can perform single activity recognition tasks, but also can recognize and locate sequential weakly labeled multi-activity data. Besides, the recurrent attention can greatly facilitate the process of sensor data accumulation by automatically segmenting the regions of interest.
BibTeX
@misc{wang2020sequential,
  author = {Kun Wang and Jun He and Lei Zhang},
  title = {{Sequential Weakly Labeled Multi-Activity Recognition and Location on Wearable Sensors using Recurrent Attention Network}},
  year = {2020},
  eprint = {2004.05768},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.05768},
}
Figure