Investigating (re)current state-of-the-art in human activity recognition datasets

Marius Bock, Alexander Hoelzemann, Michael Möller, Kristof Van LaerhovenFrontiers of Computer Science 2022 · DOI

Abstract

Human Activity Recognition (HAR) has increasingly relied on deep learning architectures like DeepConvLSTM, which combine convolutional and recurrent layers. This research investigates whether the standard two-layer LSTM configuration is truly optimal. Through an extensive ablation study across five diverse datasets, we demonstrate that a single LSTM layer frequently provides superior performance. Furthermore, we introduce the $r_{GP}$ metric to demonstrate that recurrent networks may overfit to artificial experimental protocols rather than learning human movement mechanics.

Introduction

The "de facto" standard for Human Activity Recognition (HAR) is the DeepConvLSTM, where CNN layers extract spatial features (the "shape" of a gesture) and LSTM layers capture temporal dependencies (the sequence of gestures). However, most benchmark datasets are collected under controlled laboratory conditions following strict scripts (e.g., "walk for 5 minutes, then sit for 5 minutes"). This study questions whether deep LSTMs are achieving high benchmark scores by memorizing these artificial sequences rather than learning the activities themselves—a phenomenon that would lead to failure in unpredictable real-world environments.

Methods

The authors conducted an extensive ablation study on the DeepConvLSTM architecture across five popular benchmarks: Wetlab, RWHAR, SBHAR, HHAR, and Opportunity. Three primary architectural variations were evaluated to determine the necessity of recurrent depth:

Architectural variants used in the ablation study: (A) Original 2-layer LSTM, (B) Reduced 1-layer LSTM, and (C) Convolutional-only model.
  1. The Original: 4 CNN layers followed by 2 LSTM layers.
  2. The Reduced: 4 CNN layers followed by 1 LSTM layer.
  3. The Convolutional-Only: 4 CNN layers with all recurrent layers removed.

The study also varied the number of hidden units in the LSTM layers (from 128 to 1,024) and employed Leave-One-Subject-Out (LOSO) cross-validation to ensure the models were tested on individuals not seen during training.

Results

The results challenge the "depth myth" in HAR. For four out of the five datasets, a 1-layered LSTM with 1,024 hidden units was the top performer. Increasing the width (hidden units) of a single layer proved more effective than adding a second layer, which often led to overfitting due to the small number of subjects typical in HAR datasets.

Visualization of raw accelerometer data and ground truth labels, illustrating the structured temporal patterns common in experimental protocols.

The researchers introduced a new metric, $r_{GP}$ (correlation factor), to measure the relationship between a model's generalization gap and the performance boost provided by LSTMs. High $r_{GP}$ values suggested that LSTMs were indeed picking up on "unnatural" temporal patterns. For example, in the Opportunity dataset's "Drill" sessions (highly scripted), LSTMs outperformed CNNs by 22%, whereas this advantage shrank significantly in more naturalistic "ADL" sessions.

Notably, the Wetlab dataset—which involves complex sequential DNA extraction tasks—saw the most significant improvement from recurrence, with F1-scores jumping from ~26% (0 LSTMs) to ~48% (1 LSTM), highlighting that recurrence is most valuable for complex, multi-step tasks.

Discussion and Conclusion

This study suggests that the added complexity of multiple LSTM layers is frequently unnecessary and potentially harmful in HAR tasks. Pure CNN architectures were often nearly as effective for periodical activities like walking. The findings highlight a critical need for the HAR community to move toward naturalistic data and develop regularization techniques that prevent models from "cheating" by learning experimental protocols. Future work will focus on refining the $r_{GP}$ metric to help researchers determine the necessity of recurrent layers prior to training.

Summary updated

BibTeX
@article{bock2022investigating,
  author = {Marius Bock and Alexander Hoelzemann and Michael Möller and Kristof Van Laerhoven},
  title = {{Investigating (re)current state-of-the-art in human activity recognition datasets}},
  journal = {Frontiers of Computer Science},
  year = {2022},
  doi = {10.3389/fcomp.2022.924954},
}
Figure