Abstract
Zero-shot skeleton-based action recognition aims to recognize actions of unseen categories after training on data of seen categories. The key is to build the connection between visual and semantic space from seen to unseen classes. Previous studies have primarily focused on encoding sequences into a singular feature vector, with subsequent mapping the features to an identical anchor point within the embedded space. Their performance is hindered by 1) the ignorance of the global visual/semantic distribution alignment, which results in a limitation to capture the true interdependence between the two spaces. 2) the negligence of temporal information since the frame-wise features with rich action clues are directly pooled into a single feature vector. We propose a new zero-shot skeleton-based action recognition method via mutual information (MI) estimation and maximization. Specifically, 1) we maximize the MI between visual and semantic space for distribution alignment; 2) we leverage the temporal information for estimating the MI by encouraging MI to increase as more frames are observed. Extensive experiments on three large-scale skeleton action datasets confirm the effectiveness of our method.
Introduction
Zero-shot action recognition (ZAR) represents a significant challenge in activity understanding: the ability to recognize novel actions, such as "skipping," when the model has only been exposed to different categories like "walking" or "jumping" during training. This task necessitates bridging the gap between visual movement features and semantic language representations.
Current skeleton-based ZAR methods often suffer from two primary limitations. First, they typically employ point-to-point mapping, which ignores the global distribution of visual and semantic spaces, failing to capture the complex interdependence between them. Second, they often neglect temporal dynamics by pooling entire sequences into single feature vectors, thereby losing the "story" of the movement and the relative importance of specific keyframes.
Methodology
The proposed SMIE (Skeleton-based Mutual Information Estimation) framework addresses these flaws through two core components: the Global Alignment Module and the Temporal Constraint Module.
Global Alignment Module
Instead of simple linear projections, SMIE utilizes a neural network to estimate and maximize the Mutual Information (MI) between visual and semantic distributions. By employing a Jensen-Shannon Divergence (JSD) estimator, the model learns to maximize similarity scores for paired visual-semantic data while minimizing them for unpaired negative samples. This approach aligns the underlying structure of the two data distributions, significantly enhancing generalization to unseen classes.
Temporal Constraint Module
To preserve temporal dynamics, SMIE identifies keyframes—frames with the highest motion intensity—using bidirectional motion attention. The model then generates "attention-masked" sequences where these critical frames are obscured. A hinge loss function is applied to ensure that the MI of the complete sequence remains higher than that of the masked sequence by a specific margin. This teaches the model that certain temporal segments are semantically indispensable for action identification.
Experiments and Results
The framework was evaluated on three major benchmarks: NTU-RGB+D 60, NTU-RGB+D 120, and PKU-MMD. Using standard seen/unseen splits, SMIE demonstrated superior performance over existing state-of-the-art (SOTA) methods.
- NTU-60: SMIE achieved 40.18% accuracy, representing a ~7% absolute improvement over the previous SOTA (SynSE).
- NTU-120: The model reached 45.30% accuracy, outperforming SynSE's 38.70%.
- Semantic Boosting: An innovative experiment using ChatGPT to expand simple labels into detailed descriptions resulted in a nearly 7% performance boost on NTU-60, highlighting the importance of rich semantic anchors.
Conclusion
SMIE advances zero-shot action recognition by shifting from rigid feature mapping to flexible distribution alignment via Mutual Information. While the model excels at capturing global structure and keyframe importance, challenges remain in distinguishing very short actions (e.g., "yawning") and fine-grained movements (e.g., "tearing paper") due to the inherent resolution limits of skeleton data. Future research will explore the integration of Large Language Models (LLMs) to further refine the semantic space and improve the recognition of subtle interactions.
Summary updated