Human Activity Recognition Using Recurrent Neural Networks

Deepika Singh, Erinc Merdivan, Ismini Psychoula, Johannes Kropf, Sten Hanke, Matthieu Geist, Andreas Holzinger

TL;DR

This paper demonstrates that Long Short-Term Memory (LSTM) networks can effectively classify human activities in smart homes using raw, un-engineered sensor data. By bypassing tedious manual feature extraction, the LSTM outperforms traditional probabilistic models like Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs), proving that deep learning is highly capable of capturing the complex temporal dependencies of daily routines in Ambient Assisted Living (AAL) environments.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) is a foundational technology for smart homes and Ambient Assisted Living, allowing systems to monitor and assist elderly or disabled residents. The core challenge is that data generated by simple, ubiquitous smart home sensors (like motion detectors or door switches) is often noisy and ambiguous.

Traditionally, researchers have used probabilistic models like Naive Bayes, HMMs, or CRFs to classify activities. However, Naive Bayes ignores the crucial element of time, while HMMs and CRFs often require extensive, domain-specific manual feature engineering and struggle to remember long-term dependencies (e.g., the relationship between opening a fridge and eating a meal 20 minutes later). The authors tackle this by applying a deep learning approach that can automatically learn temporal representations directly from raw data.

Methods

The authors utilize a Long Short-Term Memory (LSTM) network, a specialized type of Recurrent Neural Network (RNN) designed to model temporal sequences and avoid the vanishing gradient problem common in standard RNNs.

Instead of handcrafting features using sliding windows, the authors feed sensor data directly into the LSTM. They test the network on two distinct data representations:

  1. Raw sensor data: The direct, instantaneous binary output (0 or 1) from the sensors.
  2. Last-fired sensor data: A representation where a triggered sensor remains a "1" until a different sensor changes state, providing a rudimentary form of memory to the dataset itself.

The network was built using TensorFlow, utilizing a time slice of 70, a hidden state size of 300, and optimized using the Adam optimizer.

Datasets

The study uses three publicly available benchmark smart home datasets (originally published by Kasteren et al.).

  • Environments: Three different homes (Houses A, B, and C), each occupied by a single user.
  • Duration: Data was collected over 25, 14, and 19 days, respectively.
  • Sensors: 14 to 23 binary sensors per house, including Passive Infrared (PIR) motion detectors, pressure sensors on beds/couches, reed switches on doors/cupboards, and float sensors in toilets. Data was recorded at one-minute intervals.
  • Labels: 10 to 16 daily activities (e.g., cooking, bathing, sleeping), annotated by the users via diaries or Bluetooth headsets.

Key Insights and Ideas

  • Raw Data is Enough for Deep Learning: The most significant takeaway is that LSTMs eliminate the need for manual data preprocessing and feature handcrafting. The network successfully learned the mapping between raw sensor ticks and complex human activities, saving massive amounts of engineering time.
  • Temporal Context is King: The LSTM's ability to remember long-term dependencies allowed it to easily beat non-temporal models (like Naive Bayes) and simpler temporal models (like HMMs) on raw data.
  • Data Representation Impacts Model Advantage: Interestingly, while the LSTM dominated on raw data, traditional models like CRFs remained highly competitive when evaluated on last-fired data. Because "last-fired" data inherently bakes some temporal memory into the input, the LSTM's architectural advantage was less pronounced in that specific scenario.

Key Results

  • Raw Data Performance: The LSTM achieved the best performance across all three houses: 89.8% (House A), 85.7% (House B), and 64.2% (House C).
  • Massive Gains in Noisy Environments: In House C, the LSTM improved accuracy by roughly 40% compared to the next best model (CRF, which scored 46.3%).
  • Last-Fired Data Performance: On the last-fired datasets, the LSTM achieved 95.3% (A), 88.5% (B), and 85.9% (C). Here, it matched the top-tier performance of CRFs and Naive Bayes, though it did not drastically outperform them.
  • Computational Cost: Training took approximately 30 minutes per day of data on a single Titan X GPU, but the resulting models can be used for real-time inference without requiring a GPU.

Limitations and Future Work

The authors acknowledge a few limitations. First, there is variance in the model's predictions when training on different days, suggesting a need for better early stopping criteria and hyperparameter optimization.

More importantly, standard deep learning models like this LSTM struggle to capture model uncertainty—they cannot easily communicate when they are "unsure" of a prediction, which is a critical feature for healthcare and assisted living applications. For future work, the authors suggest exploring Bayesian Neural Networks (specifically using dropout as a Bayesian approximation) to represent uncertainty without sacrificing computational speed or accuracy.

Summary updated

Abstract

Human activity recognition using smart home sensors is one of the bases of ubiquitous computing in smart environments and a topic undergoing intense research in the field of ambient assisted living. The increasingly large amount of data sets calls for machine learning methods. In this paper, we introduce a deep learning model that learns to classify human activities without using any prior knowledge. For this purpose, a Long Short Term Memory (LSTM) Recurrent Neural Network was applied to three real world smart home datasets. The results of these experiments show that the proposed approach outperforms the existing ones in terms of accuracy and performance.
BibTeX
@inproceedings{singh2017human,
  author = {Deepika Singh and Erinc Merdivan and Ismini Psychoula and Johannes Kropf and Sten Hanke and Matthieu Geist and Andreas Holzinger},
  title = {{Human Activity Recognition Using Recurrent Neural Networks}},
  booktitle = {CD-MAKE},
  year = {2017},
  url = {https://openreview.net/forum?id=QcgTKy3KOfr},
}

Loading related content...

Figure