Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances

Shibo Zhang, Yaxuan Li, Shen Zhang, Farzad Shahabi, Stephen Xia, Yu Deng, Nabil Alshurafa

Abstract

Mobile and wearable devices have enabled numerous applications, including activity tracking, wellness monitoring, and human–computer interaction, that measure and improve our daily lives. Many of these applications are made possible by leveraging the rich collection of low-power sensors found in many mobile and wearable devices to perform human activity recognition (HAR). Recently, deep learning has greatly pushed the boundaries of HAR on mobile and wearable devices. This paper systematically categorizes and summarizes existing work that introduces deep learning methods for wearables-based HAR and provides a comprehensive analysis of the current advancements, developing trends, and major challenges. We also present cutting-edge frontiers and future directions for deep learning-based HAR.

Introduction: The Evolution of Activity Recognition

Traditional Human Activity Recognition (HAR) relied heavily on "hand-crafted" features, where engineers manually extracted mathematical properties—such as mean, variance, or frequency peaks—from sensor signals to identify specific movements. This process was labor-intensive, required significant domain expertise, and often struggled to capture the complex, non-linear patterns inherent in human motion.

With the proliferation of smartwatches, rings, and medical patches, the volume of available data has exploded, leading to two primary challenges. First, the Modeling Gap necessitates a move toward models that can learn directly from raw, noisy sensor data. Second, the Deployment Gap requires these sophisticated, power-hungry deep learning (DL) models to run efficiently on battery-constrained devices without compromising accuracy or user privacy.

A Systematic Taxonomy of HAR

To address these challenges, researchers have developed a robust taxonomy for DL-based HAR, categorizing the field by sensor types, application domains, architectural approaches, and deployment challenges.

Taxonomy of Deep Learning-based Human Activity Recognition with Wearables

Through a systematic PRISMA review process, the state-of-the-art has been organized into several key architectural families:

  • Convolutional Neural Networks (CNNs): Effective at capturing local dependencies and spatial relationships in sensor signals.
  • Recurrent Neural Networks (RNNs): Specifically LSTMs and GRUs, which are the gold standard for modeling temporal dynamics in time-series data.
  • Autoencoders (AE): Utilized for unsupervised feature learning, signal denoising, and privacy preservation.
  • Hybrid Models: Combining CNNs for feature extraction with LSTMs for temporal modeling (e.g., DeepConvLSTM) currently represents the most effective approach for complex activity recognition.
  • Generative Adversarial Networks (GANs): Used for data augmentation and generating synthetic sensor data to overcome labeling scarcity.

The application of these models is widespread, but research focus is heavily concentrated in specific areas. Fitness and lifestyle applications dominate the market, with a significant majority of research focusing on wrist-worn devices due to their high consumer adoption rates.

Distribution of application domains and sensor placement locations in HAR research

As shown in the data, the wrist accounts for nearly 60% of sensor placement in studies, while fitness and lifestyle applications comprise over 70% of the research focus. This concentration highlights a need for further exploration into other body locations and specialized medical or industrial use cases.

Benchmark Datasets and Performance

Standardized benchmarks are critical for measuring progress. Key datasets include WISDM and UCI-HAR for basic locomotion, PAMAP2 for vigorous physical activity, and OPPORTUNITY for complex daily living tasks. Deep learning has pushed performance on these benchmarks to unprecedented levels. For instance, hybrid CNN-BiLSTM models have achieved over 96% accuracy on the UCI-HAR dataset, significantly outperforming traditional machine learning methods like Support Vector Machines (SVM).

Key Insights and Technological Frontiers

  • The Hybrid Advantage: Combining the "spatial" awareness of CNNs with the "sequential" understanding of LSTMs consistently yields superior results.
  • Privacy-by-Design: Autoencoders are increasingly used to transform raw data into a "latent space" that retains activity information while stripping away sensitive identifiers like gait-based identity.
  • Edge Computing: There is a massive shift toward model compression (quantization and pruning) to allow deep models to run locally on a device's DSP or ASIC, reducing latency and cloud dependency.

Limitations and Future Directions

Despite significant advances, several roadblocks remain. The "Labeling Burden" persists, as high-quality labeled sensor data is difficult to acquire. Furthermore, generalizability remains a challenge; models trained on young, healthy subjects often fail when applied to elderly or clinical populations.

Future research is trending toward Federated Learning, which allows for training across multiple devices without sharing raw data, and Self-Supervised Learning, which could leverage the vast amounts of unlabeled wearable data to build more robust, universal activity models.

Summary updated

Abstract

Mobile and wearable devices have enabled numerous applications, including activity tracking, wellness monitoring, and human--computer interaction, that measure and improve our daily lives. Many of these applications are made possible by leveraging the rich collection of low-power sensors found in many mobile and wearable devices to perform human activity recognition (HAR). Recently, deep learning has greatly pushed the boundaries of HAR on mobile and wearable devices. This paper systematically categorizes and summarizes existing work that introduces deep learning methods for wearables-based HAR and provides a comprehensive analysis of the current advancements, developing trends, and major challenges. We also present cutting-edge frontiers and future directions for deep learning-based HAR.
BibTeX
@inproceedings{zhang2022deep,
  author = {Shibo Zhang and Yaxuan Li and Shen Zhang and Farzad Shahabi and Stephen Xia and Yu Deng and Nabil Alshurafa},
  title = {{Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=ZYIkzM3zkm},
}
Figure