Abstract
In cross-view action recognition, the appearance and motion of actions vary drastically across different viewpoints, leading to significant distribution and feature space mismatches. This paper proposes Heterogeneous Transfer Discriminant analysis of Canonical Correlations (HTDCC), a novel method to transfer action models from a source view to a target view with heterogeneous features. HTDCC learns a discriminative common feature space by simultaneously minimizing inter-class and maximizing intra-class canonical correlations. The framework includes a nonparametric criterion to reduce distribution mismatch and a joint weight learning method to fuse multiple source-view classifiers. Evaluated on the IXMAS dataset, HTDCC demonstrates superior performance in cross-view recognition tasks.
Introduction
Action recognition models often fail when the viewpoint changes between training (source) and deployment (target). This "viewpoint gap" is exacerbated by two primary factors: heterogeneous features, where different cameras provide different data types (e.g., silhouettes vs. optical flow), and the lack of paired data, where simultaneous multi-view recordings of the same instance are unavailable. HTDCC addresses these challenges by learning a shared "bridge" space that aligns disparate features without requiring paired samples.
Methodology: HTDCC
The core of HTDCC is the construction of a common feature space that preserves discriminative information while bridging the gap between source and target views.
Discriminative Common Space
Unlike standard Canonical Correlation Analysis (CCA), HTDCC utilizes label information to guide the learning process. It optimizes projection matrices to ensure that, in the shared space, samples of the same action class are clustered together (maximizing intra-class correlation) while different classes are separated (minimizing inter-class correlation).
Distribution Alignment and Weight Fusion
To further mitigate domain shift, a nonparametric criterion minimizes the distance between the means of source and target distributions. Additionally, the framework employs a Joint Weight Learning method. Recognizing that some source views are more informative for a specific target than others, the model assigns dynamic weights to each source view during fusion, prioritizing those most relevant to the target perspective.
Experimental Evaluation
The method was tested on the IXMAS (INRIA Xmas Visitor Action Dataset), featuring 11 action classes across 5 distinct camera views. To simulate heterogeneity, optical flow features were used for source views and silhouettes for target views. This setup rigorously tests the model's ability to transfer knowledge across both perspective and data-type boundaries.
Key Results
HTDCC consistently outperforms several state-of-the-art transfer learning techniques, including KCCA, HeMap, and DAMA.
As shown in the results, HTDCC achieved an average recognition accuracy of 44.6%, a substantial improvement over the 32.6% achieved by KCCA. The inclusion of distribution alignment specifically boosted pairwise performance; for instance, View 1 to View 4 transfer accuracy rose to 61.8% when using the proposed alignment criteria.
Discussion and Limitations
A key insight of this work is that mapping both views to a new, optimized common space is more effective than forcing the target to match the source feature space. However, challenges remain. In top-down views (Target View 5), actions like "get up" or "pick up" suffer from low accuracy due to self-occlusion, where the head masks body motion. Furthermore, while the iterative optimization is effective, future work is needed to scale the approach to "in-the-wild" scenarios with unconstrained viewpoints and unknown camera parameters.
Conclusion
HTDCC provides a robust framework for cross-view action recognition by handling heterogeneous features and relaxing the need for paired data. By combining discriminative manifold learning with intelligent source-view fusion, it sets a new benchmark for knowledge transfer across disparate visual perspectives.
Summary updated