Deep Residual Bidir-LSTM for Human Activity Recognition Using Wearable Sensors

Yu Zhao, Rennong Yang, Guillaume Chevalier, Maoguo GongarXiv 2017 · OpenReview

TL;DR

This paper introduces a novel deep learning architecture, Deep Residual Bidir-LSTM, for human activity recognition (HAR) using wearable sensors. By combining bidirectional LSTMs with residual connections, the model significantly improves accuracy and F1 scores on benchmark datasets, effectively addressing challenges like gradient vanishing and capturing complex temporal dependencies in human movements.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) is crucial for applications ranging from health monitoring to smart homes. However, accurately recognizing human activities from sensor data is challenging due to the complex, dynamic, and often subtle nature of human movements. Traditional machine learning methods struggle with the high dimensionality and temporal dependencies inherent in sensor data. While deep learning, particularly LSTMs, has shown promise, issues like gradient vanishing in deep networks and the need to capture both past and future contextual information for accurate prediction remain significant hurdles. This paper aims to tackle these challenges by proposing a more robust and effective deep learning architecture for HAR.

Methods

The core of this paper is the Deep Residual Bidir-LSTM network. This architecture builds upon existing deep learning components:

  • Long Short-Term Memory (LSTM): LSTMs are a type of recurrent neural network (RNN) well-suited for sequential data like time series from wearable sensors. They are designed to capture long-term dependencies and mitigate the vanishing gradient problem that plagues standard RNNs.
  • Bidirectional LSTM (Bidir-LSTM): This enhancement processes the input sequence in both forward and backward directions. This allows the model to consider both past and future context, which is vital for understanding human activities where future movements can inform the interpretation of current ones.
  • Residual Connections: Inspired by ResNet architectures in computer vision, residual connections are incorporated. These connections create "shortcuts" that allow gradients to flow more directly through the network, further combating the vanishing gradient problem, especially in very deep networks. They act as highways for gradients, enabling deeper architectures and improving the learning of complex spatial and temporal features.

The proposed architecture combines these elements by stacking bidirectional LSTM layers and integrating residual connections between them. This creates a deep network that can learn intricate patterns in sensor data while maintaining stable gradient flow.

Datasets

The authors evaluated their proposed Deep-Res-Bidir-LSTM network on two publicly available datasets:

  • Public Domain UCI Data Set: This dataset consists of activities performed by 30 volunteers aged 19-48 years, recorded using a smartphone's accelerometer and gyroscope. It includes six activities: WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS, SITTING, STANDING, and LAYING_DOWN.
  • Opportunity Data Set: This dataset is designed for benchmarking HAR algorithms and includes data from wearable, object, and ambient sensors. It features activities from four subjects, with a focus on mid-level gesture classes and a "NULL" class to handle unbalanced data.

Key Insights and Ideas

  • Bidirectionality is Key for Temporal Context: The paper highlights that human activities are continuous and often depend on both preceding and succeeding movements. Bidirectional LSTMs effectively capture this by processing information in both temporal directions, leading to more accurate predictions.
  • Residual Connections Enable Deeper, More Powerful Models: By allowing gradients to bypass layers, residual connections are crucial for training very deep networks. This enables the model to learn more complex hierarchical features from sensor data without suffering from vanishing gradients, leading to improved performance.
  • Combining Strengths: The true innovation lies in the synergistic combination of bidirectional LSTMs and residual connections. This fusion creates a powerful architecture that addresses both temporal context and the challenges of deep network training.
  • Window Size Matters: The research found that window size is a critical hyperparameter. Too small a window can break temporal continuity, while too large a window can lead to classification errors. The authors suggest a range of 500 ms to 5000 ms as generally appropriate.

Key Results

The Deep-Res-Bidir-LSTM network demonstrated significant improvements over baseline methods:

  • UCI Data Set: Achieved an accuracy increase of 4.78% compared to previously reported results. The overall accuracy reached 93.57% with an F1 score of 93.54%.
  • Opportunity Data Set: Showed an improvement of 3.68% in F1 score. The proposed algorithm achieved an F1 score of 90.20%, outperforming other algorithms tested.
  • Convergence: The F1 score trends indicated that the Deep-Res-Bidir-LSTM model converged effectively, with residual connections contributing to outstanding convergence.

Limitations and Future Work

  • Hyperparameter Tuning: The authors note that hyperparameter tuning, particularly for regularization parameters, can be challenging due to their numerousness. While grid search was used, they suggest exploring more efficient and adaptive methods for parameter tuning and network architecture evolution (e.g., automatically reshaping, adding, or removing layers).
  • Window Size Optimization: The optimal window size was found to be dataset-dependent and requires careful selection.
  • Generalization: While the model shows promise, the authors suggest applying the Deep-Res-Bidir-LSTM to other fields to assess its generalization capabilities.
  • Exploring Convolutions: Investigating the effect of mixing 1D time-based convolutions within LSTM cells is proposed as a potential avenue for further improvement.
  • Computational Resources: The paper mentions that during training, the GPU memory was sometimes exceeded, indicating potential computational demands for very large networks.

Summary updated

Abstract

Human activity recognition (HAR) has become a popular topic in research because of its wide application. With the development of deep learning, new ideas have appeared to address HAR problems. Here, a deep network architecture using residual bidirectional long short-term memory (LSTM) cells is proposed. The advantages of the new network include that a bidirectional connection can concatenate the positive time direction (forward state) and the negative time direction (backward state). Second, residual connections between stacked cells act as highways for gradients, which can pass underlying information directly to the upper layer, effectively avoiding the gradient vanishing problem. Generally, the proposed network shows improvements on both the temporal (using bidirectional cells) and the spatial (residual connections stacked deeply) dimensions, aiming to enhance the recognition rate. When tested with the Opportunity data set and the public domain UCI data set, the accuracy was increased by 4.78% and 3.68%, respectively, compared with previously reported results. Finally, the confusion matrix of the public domain UCI data set was analyzed.
BibTeX
@misc{zhao2017deep,
  author = {Yu Zhao and Rennong Yang and Guillaume Chevalier and Maoguo Gong},
  title = {{Deep Residual Bidir-LSTM for Human Activity Recognition Using Wearable Sensors}},
  year = {2017},
  eprint = {1708.08989},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.08989},
}
Figure