Introduction
Multi-view human action recognition, utilizing diverse data sources such as RGB cameras and depth sensors, offers a comprehensive understanding of human movement. However, practical implementation faces significant challenges. First, the heterogeneous gap between different modalities (e.g., RGB pixels vs. depth maps) makes effective feature fusion difficult. Second, real-world scenarios often involve incomplete data due to sensor failure or occlusion. Most existing models are brittle, experiencing sharp performance declines when a modality present during training is missing at test time.
To address these issues, the authors propose GMVAR, a framework that leverages Generative Adversarial Networks (GANs) to "hallucinate" missing data and introduces a novel View Correlation Discovery Network (VCDN) for robust multi-view fusion.
Methodology
The Generative Multi-View Action Recognition (GMVAR) framework is structured into three primary components: latent subspace learning, a generative loop for cross-view augmentation, and a correlation discovery network for final classification.
1. Latent Subspace Learning
The model maps raw RGB and Depth inputs into respective latent subspaces ($Z_1, Z_2$) using view-specific encoders ($E_1, E_2$). To ensure these representations are discriminative, a triplet loss is applied. This objective encourages the model to cluster similar actions together while separating dissimilar ones in the latent space, establishing a robust foundation for subsequent fusion.
2. Generative Loop and View Augmentation
The core innovation of GMVAR is its ability to perform "data imputation" through conditional generation. Using two conditional GANs ($G_1, G_2$), the framework can generate the latent features of a missing view from an available one. For instance, if only RGB data is available at test time, $G_2$ "hallucinates" the corresponding depth representation. Adversarial training with discriminators ($D_1, D_2$) ensures that these generated features are indistinguishable from real latent features, effectively transferring knowledge across modalities and providing robustness to "sensor death."
3. View Correlation Discovery Network (VCDN)
Standard fusion techniques, such as simple averaging or concatenation of classification scores, often fail to capture complex inter-view dependencies. The VCDN addresses this by processing the initial predictions from both view-specific classifiers ($C_1, C_2$). It constructs a cross-view discovery matrix—the outer product of the two prediction vectors—to capture high-level semantic correlations. A two-layer network then analyzes this matrix to produce the final label prediction, prioritizing the most reliable modality for specific action categories.
Experimental Evaluation
The authors evaluated GMVAR on three benchmark RGB-D datasets: Berkeley MHAD, UWA3D Multiview Activity, and DHA (Depth-included Human Action). These datasets present various challenges, including self-occlusion and diverse viewpoints.
Key Results
GMVAR consistently outperformed state-of-the-art methods, including Temporal Segment Networks (TSN) and Partial-modal GANs (PM-GANs).
- Overall Accuracy: On the DHA dataset, GMVAR achieved an accuracy of 88.72%, significantly surpassing the best baseline of 83.47%.
- Robustness to Missing Modalities: In scenarios where the model must infer RGB characteristics from depth data ($D \to R$), GMVAR showed a performance improvement of 3-7% over other generative baselines.
- Fusion Performance: Ablation studies confirmed that the VCDN fusion strategy provides a ~3% accuracy boost compared to simple score averaging or feature concatenation.
Conclusion
The GMVAR framework demonstrates that generative modeling can serve as a powerful regularizer and a solution for missing data in multi-view learning. By shifting fusion from the noisy feature space to the semantic label space via VCDN, the model achieves superior robustness and accuracy. Future work may involve scaling the architecture to handle $N$ views and exploring pixel-level generation for improved interpretability.
Summary updated