A Review on Human Activity Recognition Using Vision-Based Method

Shugang Zhang, Zhiqiang Wei, Jie Nie, Lei Huang, Shuang Wang, Zhen LiJournal of Healthcare Engineering 2017 · DOI

TL;DR

This paper provides a comprehensive review of vision-based Human Activity Recognition (HAR), tracing the evolution of representation and classification methods from global to local and depth-based approaches. It serves as a valuable resource for researchers by offering a structured taxonomy of existing literature and highlighting key challenges and future research directions in this rapidly advancing field.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) is a critical computer vision task with applications ranging from video surveillance and healthcare to human-computer interaction. The core challenge lies in accurately identifying and classifying human actions from visual data, which is complicated by factors like intraclass variation (the same activity performed differently by different people) and interclass similarity (different activities appearing visually similar). This paper addresses the need for a consolidated overview of the diverse and rapidly evolving research landscape in vision-based HAR, aiming to provide a clear roadmap for understanding current approaches and identifying future research avenues.

Methods

The paper categorizes HAR methods into two main components: representation and classification.

Representation Methods: The authors trace a chronological trajectory:

  • Global Representations: Early methods focused on representing the entire image or silhouette of a human. This includes techniques like motion energy images (MEI), motion history images (MHI), and 3D space-time volumes (STVs).
  • Local Representations: More recent approaches focus on extracting local features from interest points or dense trajectories within video frames. This often involves techniques like SIFT, HOG, HOF, and their 3D extensions, combined with bag-of-visual-words (BoVW) models for encoding.
  • Depth-Based Representations: With the advent of depth sensors (like Kinect), methods leveraging depth maps and skeletal information have emerged. This includes descriptors like Local Depth Pattern (LDP) and skeleton-based features.

Classification Methods: The paper categorizes classification approaches into:

  • Template-Based Methods: These methods compare incoming data to pre-defined templates of activities (e.g., using Dynamic Time Warping).
  • Generative Models: These models learn the joint probability distribution of data and labels (e.g., Hidden Markov Models (HMMs), Dynamic Bayesian Networks (DBNs)).
  • Discriminative Models: These models directly learn the decision boundary between classes (e.g., Support Vector Machines (SVMs), Conditional Random Fields (CRFs)).
  • Deep Learning Architectures: The paper also discusses the increasing use of deep learning, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and hybrid architectures.

Datasets

The paper reviews several representative datasets used for HAR, categorized by activity level:

  • Action Primitive Level: NTU-MSR Kinect Hand Gesture Dataset, MSRC-Kinect Gesture Dataset.
  • Action/Activity Level: KTH Activity Dataset, Weizmann Activity Dataset, MSR Action3D Dataset.
  • Interaction Level: Hollywood Human Activity Dataset, UCF Sports Dataset, MSR DailyAction3D Dataset.

The authors note that while many RGB-based datasets exist, they also highlight important recent depth or RGB-D datasets.

Key Insights and Ideas

  • Evolutionary Trajectory: HAR research has moved from holistic, global representations to more robust, localized features, and more recently, to leveraging depth information and deep learning.
  • Depth Data Revolution: The availability of low-cost depth sensors has significantly impacted HAR, enabling richer representations and more accurate recognition, especially for complex interactions.
  • Deep Learning Dominance: Deep learning architectures, particularly CNNs and RNNs, are increasingly becoming the state-of-the-art, offering powerful feature learning capabilities that surpass traditional hand-crafted methods.
  • Dataset Importance: The development of diverse and challenging benchmark datasets has been crucial for driving progress and enabling fair comparison of different HAR approaches.
  • Challenges Remain: Despite advancements, challenges like intraclass variation, interclass similarity, and real-world complexities (occlusions, varying viewpoints, dynamic backgrounds) persist.

Key Results

The paper is a review, so it doesn't present new experimental results. Instead, it synthesizes results from numerous cited works. The table of literature (Table 1) showcases a wide range of performance metrics, with accuracies often exceeding 90% for many specific tasks and datasets, particularly with recent deep learning and depth-based methods. For instance, accuracies like 95.83% on MSR Action3D dataset (Yang et al. [78]) and 96.8% on KTH dataset (Shi et al. [55]) are cited. The review highlights that the best performance is often achieved by combining multiple modalities (RGB-D) and sophisticated feature encoding methods like Fisher Vectors (FV) and Stacked Fisher Vectors (SFV).

Limitations and Future Work

Limitations: * Real-time Implementation: Many current HAR approaches are computationally intensive, making real-time implementation on resource-constrained devices (like wearables) challenging. * Dataset Specificity: Many methods are optimized for specific datasets, leading to a lack of generalizability across diverse real-world scenarios. * Low-Level Focus: Current systems often excel at recognizing basic actions/activities but struggle with higher-level, complex human behaviors and interactions.

Future Work: * Efficient Descriptors and Classifiers: Development of faster methods for feature extraction and model training to enable real-time performance. * Unified Datasets: Creating larger, more comprehensive datasets by merging existing ones to foster more generalizable algorithms. * Contextual Understanding: Moving beyond simple action recognition to understanding logical and semantic relationships between behaviors, potentially by leveraging Natural Language Processing (NLP) techniques. * Hybrid Architectures: Exploring combined ConvNet+LSTM architectures for HAR to leverage the spatial feature learning of ConvNets and the temporal modeling capabilities of LSTMs, especially for video analysis.

Summary updated

Abstract

Human activity recognition (HAR) aims to recognize activities from a series of observations on the actions of subjects and the environmental conditions. The vision-based HAR research is the basis of many applications including video surveillance, health care, and human-computer interaction (HCI). This review highlights the advances of state-of-the-art activity recognition approaches, especially for the activity representation and classification methods. For the representation methods, we sort out a chronological research trajectory from global representations to local representations, and recent depth-based representations. For the classification methods, we conform to the categorization of template-based methods, discriminative models, and generative models and review several prevalent methods. Next, representative and available datasets are introduced. Aiming to provide an overview of those methods and a convenient way of comparing them, we classify existing literatures with a detailed taxonomy including representation and classification methods, as well as the datasets they used. Finally, we investigate the directions for future research.
BibTeX
@article{zhang2017review,
  author = {Shugang Zhang and Zhiqiang Wei and Jie Nie and Lei Huang and Shuang Wang and Zhen Li},
  title = {{A Review on Human Activity Recognition Using Vision-Based Method}},
  journal = {Journal of Healthcare Engineering},
  year = {2017},
  doi = {10.1155/2017/3090343},
}
Figure