A Hybrid Network for Large-Scale Action Recognition from RGB and Depth Modalities

Huogen Wang, Zhanjie Song, Wanqing Li, Pichao Wang

TL;DR

This paper introduces a novel hybrid deep learning network for action recognition that effectively combines RGB and depth data. By introducing "weighted dynamic images" and a fusion strategy using Canonical Correlation Analysis (CCA), the method achieves state-of-the-art results on challenging large-scale datasets, significantly outperforming existing approaches.

What Problem Does This Paper Address?

Recognizing human actions from video is a complex task, especially when dealing with large-scale datasets that involve diverse actions and subtle movements. Existing methods often struggle to capture both the spatial appearance information (from RGB) and the geometric, illumination-invariant information (from depth) effectively. Furthermore, different types of actions require different combinations of spatial, temporal, and structural information for accurate recognition. For instance, actions involving object interaction (like "putting on a jacket") are harder to distinguish than simple body movements ("standing up"). Current approaches often have limitations in adaptively leveraging these different types of information, leading to performance drops on certain action categories, particularly those involving object interactions or subtle, short-duration movements.

Methods

The core of this work is a hybrid network that combines the strengths of several deep learning architectures.

  • Weighted Dynamic Images (WDIs): This is a novel representation that extends traditional "dynamic images." Instead of simply averaging frames, WDIs use a "weighted rank pooling" mechanism. This mechanism assigns different importance (weights) to individual frames and even specific regions within frames, allowing the network to focus on more discriminative parts of the action. This is achieved through a bidirectional approach, generating both forward and backward WDIs.
  • CNN-based Component: The WDIs (both RGB and depth) are fed into separate ResNet-50 CNNs to extract rich spatial and spatio-temporal features.
  • 3D ConvLSTM Component: The raw RGB and depth video sequences are also processed by 3D Convolutional LSTM (ConvLSTM) networks. This component is designed to capture short-term spatio-temporal features using 3D convolutions and then model long-term temporal dynamics with LSTMs.
  • Canonical Correlation Analysis (CCA) Fusion: To effectively combine the features extracted from the CNNs and 3D ConvLSTMs, the authors employ CCA. This method is chosen over simple concatenation because it can reduce redundancy and dimensionality while preserving discriminative information, which is crucial when fusing features from multiple modalities.
  • Linear SVM Classifier: The fused features from CCA are then fed into a linear Support Vector Machine (SVM) for the final action classification.

Datasets

The proposed method was evaluated on three large-scale, publicly available datasets:

  • ChaLearn LAP IsoGD Dataset: This dataset focuses on isolated gestures and includes both RGB and depth data.
  • NTU RGB+D Dataset: A widely used, large-scale dataset for human activity analysis, containing RGB, depth, and skeleton data across numerous action classes.
  • Multi-modal & Multi-view & Interactive (M²I) Dataset: This dataset captures human-object and human-human interactions.

Key Insights and Ideas

  • Weighted Dynamic Images are More Discriminative: The paper convincingly shows that their proposed Weighted Dynamic Images (WDIs) are superior to traditional Dynamic Images (DIs). WDIs can better capture subtle but crucial aspects of actions, like hand movements during object interaction, which are often missed by standard DIs.
  • Modality Fusion is Crucial for Complex Actions: The success of the hybrid approach, combining RGB and depth, highlights that neither modality alone is sufficient for robust action recognition, especially for actions involving object interactions.
  • CCA is Effective for Feature Fusion: Using Canonical Correlation Analysis for feature fusion proves to be a powerful technique. It effectively merges information from different network components and modalities without introducing excessive redundancy, leading to better classification performance.
  • Adaptive Feature Weighting Matters: The core idea of assigning weights to frames and regions within WDIs is key. This adaptive weighting allows the model to focus on the most informative parts of an action, improving its ability to distinguish between similar actions or actions with subtle discriminative cues.

Key Results

The proposed hybrid network achieved state-of-the-art results on all three tested datasets.

  • On the ChaLearn LAP IsoGD Dataset, the fusion of WDDI and WDRI achieved 55.64% accuracy, a significant improvement over existing methods.
  • On the NTU RGB+D Dataset, the method achieved 89.51% accuracy for RGB+Depth fusion in the cross-subject protocol and 91.68% in the cross-view protocol, outperforming skeleton-based and other RGB+D methods. Notably, it achieved over 12 percentage points improvement on category C4 actions (those involving object interaction) compared to ConvLSTM and DI+CNN.
  • On the M²I Dataset, the proposed method achieved 100% accuracy in the single-task scenario and 93.8% / 90.6% in the cross-view scenario, surpassing all other reported methods.

Limitations and Future Work

  • Computational Cost: While achieving high accuracy, the hybrid nature of the network, combining CNNs and 3D ConvLSTMs, likely incurs significant computational cost for training and inference.
  • Skeleton Modality Extension: The authors explicitly mention that the proposed network can be extended to include the skeleton modality, which could further enhance performance, especially for actions where human pose is a primary indicator.
  • Learning Weights: The paper discusses potential ways to compute spatial and temporal weights for WDIs but notes that learning these weights is beyond the scope of the current work. Future research could explore end-to-end learning of these weights.

Summary updated

Abstract

The paper presents a novel hybrid network for large-scale action recognition from multiple modalities. The network is built upon the proposed weighted dynamic images. It effectively leverages the strengths of the emerging Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) based approaches to specifically address the challenges that occur in large-scale action recognition and are not fully dealt with by the state-of-the-art methods. Specifically, the proposed hybrid network consists of a CNN based component and an RNN based component. Features extracted by the two components are fused through canonical correlation analysis and then fed to a linear Support Vector Machine (SVM) for classification. The proposed network achieved state-of-the-art results on the ChaLearn LAP IsoGD, NTU RGB+D and Multi-modal & Multi-view & Interactive ( M 2 I ) datasets and outperformed existing methods by a large margin (over 10 percentage points in some cases).
BibTeX
@inproceedings{wang2020hybrid,
  author = {Huogen Wang and Zhanjie Song and Wanqing Li and Pichao Wang},
  title = {{A Hybrid Network for Large-Scale Action Recognition from RGB and Depth Modalities}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=08L1Z0InC8e},
}
Figure