Cross-View Action Recognition over Heterogeneous Feature Spaces

Xinxiao Wu, Han Wang, Cuiwei Liu, Yunde Jia

Abstract

In cross-view action recognition, the appearance and motion of actions vary drastically across different viewpoints, leading to significant distribution and feature space mismatches. This paper proposes Heterogeneous Transfer Discriminant analysis of Canonical Correlations (HTDCC), a novel method to transfer action models from a source view to a target view with heterogeneous features. HTDCC learns a discriminative common feature space by simultaneously minimizing inter-class and maximizing intra-class canonical correlations. The framework includes a nonparametric criterion to reduce distribution mismatch and a joint weight learning method to fuse multiple source-view classifiers. Evaluated on the IXMAS dataset, HTDCC demonstrates superior performance in cross-view recognition tasks.

Introduction

Action recognition models often fail when the viewpoint changes between training (source) and deployment (target). This "viewpoint gap" is exacerbated by two primary factors: heterogeneous features, where different cameras provide different data types (e.g., silhouettes vs. optical flow), and the lack of paired data, where simultaneous multi-view recordings of the same instance are unavailable. HTDCC addresses these challenges by learning a shared "bridge" space that aligns disparate features without requiring paired samples.

Methodology: HTDCC

The core of HTDCC is the construction of a common feature space that preserves discriminative information while bridging the gap between source and target views.

Discriminative Common Space

Unlike standard Canonical Correlation Analysis (CCA), HTDCC utilizes label information to guide the learning process. It optimizes projection matrices to ensure that, in the shared space, samples of the same action class are clustered together (maximizing intra-class correlation) while different classes are separated (minimizing inter-class correlation).

Distribution Alignment and Weight Fusion

To further mitigate domain shift, a nonparametric criterion minimizes the distance between the means of source and target distributions. Additionally, the framework employs a Joint Weight Learning method. Recognizing that some source views are more informative for a specific target than others, the model assigns dynamic weights to each source view during fusion, prioritizing those most relevant to the target perspective.

Multi-source view weighting framework showing how different source views contribute to target view recognition.

Experimental Evaluation

The method was tested on the IXMAS (INRIA Xmas Visitor Action Dataset), featuring 11 action classes across 5 distinct camera views. To simulate heterogeneity, optical flow features were used for source views and silhouettes for target views. This setup rigorously tests the model's ability to transfer knowledge across both perspective and data-type boundaries.

Key Results

HTDCC consistently outperforms several state-of-the-art transfer learning techniques, including KCCA, HeMap, and DAMA.

Recognition accuracy comparison of HTDCC against state-of-the-art transfer learning methods.

As shown in the results, HTDCC achieved an average recognition accuracy of 44.6%, a substantial improvement over the 32.6% achieved by KCCA. The inclusion of distribution alignment specifically boosted pairwise performance; for instance, View 1 to View 4 transfer accuracy rose to 61.8% when using the proposed alignment criteria.

Discussion and Limitations

A key insight of this work is that mapping both views to a new, optimized common space is more effective than forcing the target to match the source feature space. However, challenges remain. In top-down views (Target View 5), actions like "get up" or "pick up" suffer from low accuracy due to self-occlusion, where the head masks body motion. Furthermore, while the iterative optimization is effective, future work is needed to scale the approach to "in-the-wild" scenarios with unconstrained viewpoints and unknown camera parameters.

Conclusion

HTDCC provides a robust framework for cross-view action recognition by handling heterogeneous features and relaxing the need for paired data. By combining discriminative manifold learning with intelligent source-view fusion, it sets a new benchmark for knowledge transfer across disparate visual perspectives.

Summary updated

Abstract

In cross-view action recognition, "what you saw" in one view is different from "what you recognize" in another view. The data distribution even the feature space can change from one view to another due to the appearance and motion of actions drastically vary across different views. In this paper, we address the problem of transferring action models learned in one view (source view) to another different view (target view), where action instances from these two views are represented by heterogeneous features. A novel learning method, called Heterogeneous Transfer Discriminantanalysis of Canonical Correlations (HTDCC), is proposed to learn a discriminative common feature space for linking source and target views to transfer knowledge between them. Two projection matrices that respectively map data from source and target views into the common space are optimized via simultaneously minimizing the canonical correlations of inter-class samples and maximizing the intraclass canonical correlations. Our model is neither restricted to corresponding action instances in the two views nor restricted to the same type of feature, and can handle only a few or even no labeled samples available in the target view. To reduce the data distribution mismatch between the source and target views in the common feature space, a nonparametric criterion is included in the objective function. We additionally propose a joint weight learning method to fuse multiple source-view action classifiers for recognition in the target view. Different combination weights are assigned to different source views, with each weight presenting how contributive the corresponding source view is to the target view. The proposed method is evaluated on the IXMAS multi-view dataset and achieves promising results.
BibTeX
@inproceedings{wu2013cross,
  author = {Xinxiao Wu and Han Wang and Cuiwei Liu and Yunde Jia},
  title = {{Cross-View Action Recognition over Heterogeneous Feature Spaces}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  url = {https://openreview.net/forum?id=S1Z4sezO-H},
}
Figure