RGB-D-based action recognition datasets: A survey

Jing Zhang, Wanqing Li, Philip O. Ogunbona, Pichao Wang, Chang Tang

This survey paper provides a comprehensive roadmap of the RGB-D (Red, Green, Blue, and Depth) action recognition landscape. It catalogs and analyzes 44 benchmark datasets, critiques the lack of standardized evaluation protocols, and highlights the growing gap between "saturated" laboratory results and real-world performance.

TL;DR

The paper provides a much-needed taxonomy of 44 RGB-D action recognition datasets, categorizing them by view-type and complexity. It reveals that while many algorithms are achieving near-perfect accuracy on lab-captured data, the field suffers from inconsistent evaluation protocols and a lack of "in-the-wild" data necessary for real-world deployment.

What Problem Does This Paper Address?

Since the release of the Microsoft Kinect in 2010, the field of human action recognition has shifted from 2D video to 3D RGB-D data. This shift led to a "gold rush" of new datasets and algorithms. However, this rapid growth created a fragmented landscape.

Researchers faced several challenges:

  1. Dataset Selection: With dozens of options, it was unclear which dataset best modeled specific applications (e.g., healthcare vs. gaming).
  2. Inconsistent Evaluation: There was no "standard" way to test an algorithm. One paper might use a "cross-subject" split, while another used "leave-one-out," making direct comparisons impossible.
  3. The Saturation Trap: Many datasets were recorded in controlled lab settings, leading to inflated accuracy scores that failed when applied to messy, real-world environments.

Methods

As a survey paper, the authors employ a systematic analysis rather than a new algorithm. They categorize the 44 reviewed datasets into three primary groups:

  • Single-view: Actions performed by one person from one fixed camera angle (27 datasets).
  • Multi-view: Actions captured from multiple angles or heights simultaneously (10 datasets).
  • Multi-person/Interaction: Activities involving two or more people interacting (7 datasets).

The authors also define four Acquisition Modes (ranging from simple pre-segmented action clips to complex, random sequences of actions) and three levels of Environmental Complexity (Low, Medium, High) based on background clutter and occlusion.

Datasets

The paper reviews 44 datasets in total. Some of the most influential include:

  • MSR-Action3D: The pioneer dataset that set the baseline for the field.
  • CAD-60 & CAD-120: Focused on daily activities in home/office environments with human-object interaction.
  • Berkeley MHAD: A highly complex multimodal dataset including depth, accelerometers, and microphones.
  • SBU Kinect Interaction: The standard for recognizing two-person interactions like hugging or punching.
  • UTKinect & RGBD-HuDaAct: Early benchmarks for single-view daily activity recognition.

Key Insights and Ideas

  • The "Saturated" State: Several popular datasets (like MSR-Action3D) have reached nearly 100% accuracy in published literature. The authors argue these datasets are now "saturated" and no longer useful for pushing the state-of-the-art.
  • Protocol Chaos: The authors find that "Cross-Subject" testing—the most common protocol—is implemented differently across papers (e.g., different ratios of training vs. testing subjects), leading to "unfair and objective comparative evaluation."
  • The Modality Trade-off: While skeleton data (joint positions) is popular because it's computationally "cheap," it often misses crucial context provided by the RGB or Depth maps, such as the objects a person is holding.
  • Context Matters: The survey emphasizes that an action like "drinking" looks very different depending on whether the subject is sitting on a sofa (Daily Activity) or standing in front of a gaming console (HCI).

Key Results

The paper tabulates the State-of-the-Art (SOTA) results for all 44 datasets.

  • Single-View: Accuracy is extremely high, often exceeding 95-98% on datasets like MSR-Action3D and UTKinect.
  • Multi-View: Performance drops significantly when algorithms are asked to recognize an action from an angle they haven't seen during training (Cross-view).
  • Multi-Person: This remains the most challenging frontier, with lower average accuracies and higher sensitivity to occlusion (people blocking each other).

Limitations and Future Work

The authors identify several critical bottlenecks for the field:

  • Dataset Size: Compared to 2D video datasets (which have millions of samples), RGB-D datasets are still tiny, limited by the need for manual capture in labs.
  • Lack of Camera Motion: Almost all current datasets use fixed cameras. Real-world robots or mobile phones will have moving cameras, which current algorithms aren't trained to handle.
  • Standardization: The authors strongly recommend that the community adopt "standard" and unified evaluation protocols and include statistical significance tests in their reports.
  • Cross-Dataset Testing: To prove an algorithm is truly robust, it should be trained on one dataset and tested on a completely different one—a practice that was virtually non-existent at the time of this survey.

Summary updated

Abstract

Highlights • A detailed review and in-depth analysis of 44 publicly available RGB-D-based action datasets. • Recommendations on the selection of datasets and evaluation protocols for use in future research. • Identification of some limitations of these datasets and evaluation protocols. • Recommendations on future creation of datasets and use of evaluation protocols. Abstract Human action recognition from RGB-D (Red, Green, Blue and Depth) data has attracted increasing attention since the first work reported in 2010. Over this period, many benchmark datasets have been created to facilitate the development and evaluation of new algorithms. This raises the question of which dataset to select and how to use it in providing a fair and objective comparative evaluation against state-of-the-art methods. To address this issue, this paper provides a comprehensive review of the most commonly used action recognition related RGB-D video datasets, including 27 single-view datasets, 10 multi-view datasets, and 7 multi-person datasets. The detailed information and analysis of these datasets is a useful resource in guiding insightful selection of datasets for future research. In addition, the issues with current algorithm evaluation vis-á-vis limitations of the available datasets and evaluation protocols are also highlighted; resulting in a number of recommendations for collection of new datasets and use of evaluation protocols.
BibTeX
@article{zhang2016rgb,
  author = {Jing Zhang and Wanqing Li and Philip O. Ogunbona and Pichao Wang and Chang Tang},
  title = {{RGB-D-based action recognition datasets: A survey}},
  journal = {Pattern Recognition},
  year = {2016},
  url = {https://openreview.net/forum?id=0jPjXg9tyW3},
}
Figure