Activity recognition in unconstrained RGB-D video using 3D trajectories

Yang Xiao, Gangqiang Zhao, Junsong Yuan, Daniel ThalmannSIGGRAPH ASIA Autonomous Virtual Humans and Social Robot for Telepresence 2014 · DOI

TL;DR

This paper introduces a novel approach to recognize human activities in unconstrained RGB-D videos by extending 2D dense trajectories into the third dimension. By leveraging depth information, their method captures crucial motion along the depth axis, leading to state-of-the-art performance on a challenging benchmark dataset.

What Problem Does This Paper Address?

Recognizing human activities in videos is a long-standing challenge, especially in "unconstrained" scenarios. These videos often suffer from cluttered backgrounds, camera motion, and varying viewpoints, making it difficult for algorithms to distinguish meaningful human actions. While existing methods using RGB video have shown promise, they often ignore the rich motion information available in the depth dimension provided by RGB-D sensors (like Microsoft Kinect). This depth information is crucial for a more robust and accurate understanding of human movement, but effectively integrating it into activity recognition pipelines remains an open problem.

Methods

The core innovation of this paper is the extension of 2D dense trajectories into 3D trajectories. Here's how they do it:

  1. 2D Dense Trajectory Extraction: They start by using standard techniques to extract dense trajectories in the 2D RGB video. This involves sampling feature points densely in each frame and tracking them using optical flow.
  2. Depth Mapping and Scene Flow Estimation: The key step is mapping the 2D trajectory points from the RGB frames to their corresponding locations in the depth frames. Using the depth values at these mapped locations, they estimate the "scene flow" along the depth direction (Vz). This Vz, combined with the 2D optical flow (Vx, Vy), forms a 3D trajectory for each tracked point.
  3. Handling Camera Motion: To ensure the trajectories represent human motion rather than camera movement, they employ a verification step using rectified optical flow and spatially smooth the depth map to reduce noise.
  4. Feature Extraction: To describe these 3D trajectories, they propose two novel descriptors:
  5. MBHz (Motion Boundary Histogram in depth): An extension of the successful MBH descriptor, applied to the depth motion component. This is particularly robust to camera motion along the depth axis.
  6. Trajectory Shape Descriptor (Trz): Captures the shape of the trajectory in the depth dimension. These descriptors, along with existing ones like MBHx and MBHy, are then concatenated to form a comprehensive feature representation.

  7. Classification: Finally, they use Fisher Vector encoding to create a global representation of the video and a linear Support Vector Machine (SVM) for classification.

Datasets

  • Hollywood 3D Dataset: This is the primary dataset used for evaluation. It's a challenging benchmark for unconstrained RGB-D activity recognition, featuring 14 action classes and including significant camera motion and viewpoint changes. The dataset's depth data is reconstructed, which the authors note can impact quality.

Key Insights and Ideas

  • Depth is Crucial for Unconstrained Activity Recognition: The paper strongly suggests that ignoring depth information is a significant limitation for current RGB-based activity recognition methods, especially in complex scenarios.
  • 3D Trajectories Offer a More Complete Motion Representation: Extending 2D trajectories to 3D provides a richer understanding of how objects and people move in space, capturing motion along the depth axis which is vital for distinguishing actions.
  • MBHz is a Powerful Descriptor for Depth Motion: The proposed MBHz descriptor proves effective in characterizing motion along the depth direction, offering robustness against camera motion that can confound other methods.
  • Trajectory-Based Methods Outperform Traditional STIPs: The results highlight that trajectory-based approaches (both 2D and 3D) are generally more suitable for unconstrained activity recognition compared to traditional spatio-temporal interest point (STIP) based methods.

Key Results

  • State-of-the-Art Performance: The proposed 3D trajectory approach, particularly when using the combined MBHxyz descriptor, achieved the state-of-the-art performance on the Hollywood 3D dataset, outperforming baseline methods like 3.5D Harris STIP by a significant margin.
  • CCR and AP Improvements: For example, with MBH/MBHy/MBHz, 3D trajectories achieved a Correct Classification Rate (CCR) of 29.76% and an Average Precision (AP) of 33.17%, significantly better than the 21.80% CCR and 14.10% AP of 3.5D Harris STIP with HOG/HOF/HoDG.
  • Descriptor Effectiveness: MBH (combining all three dimensions) was found to be the most robust descriptor. Combining trajectory shape descriptors (Trxyz) with MBHxyz further boosted performance, reaching a CCR of 31.14%.

Limitations and Future Work

  • Depth Data Quality: The authors acknowledge that the depth data in the Hollywood 3D dataset is reconstructed, not directly captured by depth sensors. This limitation might explain why the AP of 3D trajectories was sometimes lower than 2D dense trajectories.
  • Future Dataset Construction: To further validate their method, they plan to construct a new unconstrained RGB-D activity dataset with directly captured depth data and camera motion.
  • Deep Learning Integration: They also express interest in exploring how feature learning technologies, such as Convolutional Neural Networks (CNNs), can be used to further enhance RGB-D activity recognition performance.

Summary updated

BibTeX
@inproceedings{xiao2014activity,
  author = {Yang Xiao and Gangqiang Zhao and Junsong Yuan and Daniel Thalmann},
  title = {{Activity recognition in unconstrained RGB-D video using 3D trajectories}},
  booktitle = {SIGGRAPH ASIA Autonomous Virtual Humans and Social Robot for Telepresence},
  year = {2014},
  doi = {10.1145/2668956.2668961},
}
Figure