Viewpoint Invariant Action Recognition using RGB-D Videos

Jian Liu, Naveed Akhtar, Ajmal S. Mian

TL;DR

This paper presents a robust framework for viewpoint-invariant action recognition by fusing complementary information from RGB and Depth video streams. The authors combine a non-linear knowledge transfer model for RGB data with a CNN-based human pose model for Depth data, topped off with a novel "sparse-dense" collaborative classification scheme that significantly outperforms existing state-of-the-art methods.

What Problem Does This Paper Address?

Action recognition is a cornerstone of computer vision, but it faces a persistent hurdle: viewpoint variation. A "waving" action looks fundamentally different when filmed from the side versus from above. While Depth sensors (like the Kinect) have helped by providing 3D structure that is less sensitive to lighting and texture, Depth data itself is still highly sensitive to the camera's angle.

Existing methods often struggle to bridge this gap. Some rely on skeleton joint estimation, which is prone to error in real-world conditions. Others use either RGB or Depth in isolation, failing to capitalize on how these modalities complement each other—RGB provides high-fidelity texture and motion cues, while Depth excels at capturing human pose and shape. The authors aim to create a system that is truly "view-invariant" by extracting features that remain consistent regardless of the camera's position.

Methods

The proposed approach uses a dual-stream pipeline to extract and then fuse heterogeneous features.

Figure 1: Schematics of the proposed approach: During training, the Depth stream extracts

1. The Depth Stream

The system uses a pre-trained Human Pose Model (HPM), a CNN architecture similar to AlexNet but trained on synthetic data from 180 different viewpoints. This makes the resulting features inherently robust to angle changes. To capture how the pose changes over time, the authors apply a Fourier Temporal Pyramid (FTP) to the CNN features, resulting in a high-dimensional spatio-temporal descriptor.

2. The RGB Stream

For the RGB data, the authors extract dense trajectories to capture motion. These are encoded using a representative codebook. To achieve viewpoint invariance here, they employ a Non-linear Knowledge Transfer Model (NKTM). This deep network is trained to map features from any arbitrary viewpoint to a single "canonical" (frontal) viewpoint, effectively "straightening out" the perspective before classification.

3. Feature Fusion and Classification

Because the Depth and RGB features have vastly different scales and dimensions, the authors use Z-score normalization and rescaling before concatenating them into a single heterogeneous feature vector.

The final classification step is a novel Sparse-Dense Collaborative Representation (CR) scheme. Standard CR-based classifiers usually favor either a sparse representation (using only a few training samples to represent the test sample) or a dense one (using many). The authors propose a convex combination of both, arguing that this hybrid approach better handles the complex, heterogeneous nature of fused RGB-D features.

Datasets

The authors evaluated their method on three major benchmarks:

  • UWA 3D Multiview Activity II Dataset: 30 actions performed by 10 subjects, recorded from 4 distinct viewpoints (front, left, right, top).
  • Northwestern-UCLA Multiview Action 3D Dataset: 10 action classes captured by 3 Kinect cameras simultaneously.
  • NTU RGB+D Human Activity Dataset: A massive, large-scale dataset featuring 60 action types and over 56,000 samples, including complex cross-view and cross-subject evaluation protocols.

Key Insights and Ideas

  • Late Fusion is Key: Instead of combining RGB and Depth data early in the pipeline, the authors process them individually to fully capitalize on their unique characteristics (motion/texture vs. pose/shape) before fusing them at the feature level.
  • Canonical Mapping: The use of the NKTM to project all RGB features into a single canonical viewpoint is a powerful way to handle the "infinite" variety of camera angles without needing to see every angle during training.
  • Balancing Sparsity and Density: The insight that fused features might benefit from both sparse and dense representations led to a more flexible classifier that achieves global optimality regarding its combination parameter ($\lambda_1$).

Key Results

The proposed method demonstrated significant performance gains across all datasets:

  • Superior Accuracy: On the UWA 3D-II dataset, the method achieved an average accuracy of 84.6%, a 7.7% improvement over its nearest competitor (HPM+TM).
  • Robustness to Large Variations: The most significant improvements were seen in scenarios with large viewpoint shifts (e.g., training on side views and testing on a top view), where the error rate was reduced by up to 36.4%.
  • Large-Scale Success: On the massive NTU RGB+D dataset, the method achieved 84.5% accuracy in cross-view testing, outperforming even many specialized skeleton-based methods.

Limitations and Future Work

While the paper demonstrates impressive results, a few areas remain for exploration:

  • Computational Complexity: The feature vectors are extremely high-dimensional (over 120,000 dimensions), which may pose challenges for real-time applications without significant hardware acceleration.
  • End-to-End Learning: The current pipeline involves several discrete steps (trajectory extraction, codebook encoding, NKTM, HPM). Future work could investigate a more unified, end-to-end deep learning architecture that might further simplify the process.

Summary updated

Abstract

In video-based action recognition, viewpoint variations often pose major challenges because the same actions can appear different from different views. We use the complementary RGB and Depth information from the RGB-D cameras to address this problem. The proposed technique capitalizes on the spatio-temporal information available in the two data streams to the extract action features that are largely insensitive to the viewpoint variations. We use the RGB data to compute dense trajectories that are translated to viewpoint insensitive deep features under a non-linear knowledge transfer model. Similarly, the Depth stream is used to extract CNN-based view invariant features on which Fourier Temporal Pyramid is computed to incorporate the temporal information. The heterogeneous features from the two streams are combined and used as a dictionary to predict the label of the test samples. To that end, we propose a sparse-dense collaborative representation classification scheme that strikes a balance between the discriminative abilities of the dense and the sparse representations of the samples over the extracted heterogeneous dictionary.
BibTeX
@inproceedings{liu2017viewpoint,
  author = {Jian Liu and Naveed Akhtar and Ajmal S. Mian},
  title = {{Viewpoint Invariant Action Recognition using RGB-D Videos}},
  booktitle = {IEEE Access},
  year = {2017},
  doi = {10.1109/access.2018.2880231},
}
Figure