Generative Multi-View Human Action Recognition

Lichen Wang, Zhengming Ding, Zhiqiang Tao, Yunyu Liu, Yun Fu

Introduction

Multi-view human action recognition, utilizing diverse data sources such as RGB cameras and depth sensors, offers a comprehensive understanding of human movement. However, practical implementation faces significant challenges. First, the heterogeneous gap between different modalities (e.g., RGB pixels vs. depth maps) makes effective feature fusion difficult. Second, real-world scenarios often involve incomplete data due to sensor failure or occlusion. Most existing models are brittle, experiencing sharp performance declines when a modality present during training is missing at test time.

To address these issues, the authors propose GMVAR, a framework that leverages Generative Adversarial Networks (GANs) to "hallucinate" missing data and introduces a novel View Correlation Discovery Network (VCDN) for robust multi-view fusion.

Methodology

The Generative Multi-View Action Recognition (GMVAR) framework is structured into three primary components: latent subspace learning, a generative loop for cross-view augmentation, and a correlation discovery network for final classification.

The GMVAR architecture, featuring view-specific encoders, a generative loop for missing modality hallucination, and the View Correlation Discovery Network (VCDN).

1. Latent Subspace Learning

The model maps raw RGB and Depth inputs into respective latent subspaces ($Z_1, Z_2$) using view-specific encoders ($E_1, E_2$). To ensure these representations are discriminative, a triplet loss is applied. This objective encourages the model to cluster similar actions together while separating dissimilar ones in the latent space, establishing a robust foundation for subsequent fusion.

2. Generative Loop and View Augmentation

The core innovation of GMVAR is its ability to perform "data imputation" through conditional generation. Using two conditional GANs ($G_1, G_2$), the framework can generate the latent features of a missing view from an available one. For instance, if only RGB data is available at test time, $G_2$ "hallucinates" the corresponding depth representation. Adversarial training with discriminators ($D_1, D_2$) ensures that these generated features are indistinguishable from real latent features, effectively transferring knowledge across modalities and providing robustness to "sensor death."

3. View Correlation Discovery Network (VCDN)

Standard fusion techniques, such as simple averaging or concatenation of classification scores, often fail to capture complex inter-view dependencies. The VCDN addresses this by processing the initial predictions from both view-specific classifiers ($C_1, C_2$). It constructs a cross-view discovery matrix—the outer product of the two prediction vectors—to capture high-level semantic correlations. A two-layer network then analyzes this matrix to produce the final label prediction, prioritizing the most reliable modality for specific action categories.

Experimental Evaluation

The authors evaluated GMVAR on three benchmark RGB-D datasets: Berkeley MHAD, UWA3D Multiview Activity, and DHA (Depth-included Human Action). These datasets present various challenges, including self-occlusion and diverse viewpoints.

Key Results

GMVAR consistently outperformed state-of-the-art methods, including Temporal Segment Networks (TSN) and Partial-modal GANs (PM-GANs).

  • Overall Accuracy: On the DHA dataset, GMVAR achieved an accuracy of 88.72%, significantly surpassing the best baseline of 83.47%.
  • Robustness to Missing Modalities: In scenarios where the model must infer RGB characteristics from depth data ($D \to R$), GMVAR showed a performance improvement of 3-7% over other generative baselines.
  • Fusion Performance: Ablation studies confirmed that the VCDN fusion strategy provides a ~3% accuracy boost compared to simple score averaging or feature concatenation.
Accuracy comparison over training epochs, demonstrating the superior performance of the VCDN approach compared to single-view and standard fusion baselines.

Conclusion

The GMVAR framework demonstrates that generative modeling can serve as a powerful regularizer and a solution for missing data in multi-view learning. By shifting fusion from the noisy feature space to the semantic label space via VCDN, the model achieves superior robustness and accuracy. Future work may involve scaling the architecture to handle $N$ views and exploring pixel-level generation for improved interpretability.

Summary updated

Abstract

Multi-view action recognition targets to integrate complementary information from different views to improve classification performance. It is a challenging task due to the distinct gap between heterogeneous feature domains. Moreover, most existing methods neglect to consider the incomplete multi-view data, which limits their potential compatibility in real-world applications. In this work, we propose a Generative Multi-View Action Recognition (GMVAR) framework to address the challenges above. The adversarial generative network is leveraged to generate one view conditioning on the other view, which fully explores the latent connections in both intra-view and cross-view aspects. Our approach enhances the model robustness by employing adversarial training, and naturally handles the incomplete view case by imputing the missing data. Moreover, an effective View Correlation Discovery Network (VCDN) is proposed to further fuse the multi-view information in a higher-level label space. Extensive experiments demonstrate the effectiveness of our proposed approach by comparing with state-of-the-art algorithms.
BibTeX
@inproceedings{wang2019generative,
  author = {Lichen Wang and Zhengming Ding and Zhiqiang Tao and Yunyu Liu and Yun Fu},
  title = {{Generative Multi-View Human Action Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00631},
}
Figure