Abstract
Skeleton-based action recognition models often struggle with real-world "long-tailed" data distributions, where common actions are overrepresented and critical rare actions are scarce. Standard augmentation techniques like Mixup often create semantically confusing samples for sparse skeleton data. We introduce Shap-Mix, a data augmentation framework that utilizes Shapley values to identify salient body parts for rare actions. By ensuring these key motion patterns are preserved during the mixing process, Shap-Mix generates representative synthetic samples that effectively regularize the decision boundaries for tail classes. Our method achieves state-of-the-art results on long-tailed versions of NTU RGB+D and Kinetics-Skeleton datasets.
Introduction: The Long-Tail Challenge
In practical applications, human action recognition datasets are rarely balanced. Models typically perform well on "head" classes (e.g., standing, walking) but fail on "tail" classes (e.g., falling, seizure). Skeleton-based models, which represent humans as a series of 3D keypoints, are particularly sensitive to this imbalance. Traditional mixing strategies often lead to "semantic confusion"—synthetic samples that lack physical or logical coherence, thereby degrading the model's ability to learn distinct features for rare actions.
Methodology
The Shap-Mix framework addresses these challenges through a two-stage process: spatial-temporal mixing and Shapley-guided saliency preservation.
Spatial-Temporal Mixing (ST-Mix)
To maintain structural integrity, we move beyond joint-level mixing. ST-Mix operates at the body-part level, dividing the skeleton into five semantic regions (trunk, arms, and legs). Temporally, it employs a down-sampling strategy to preserve the full progression of an action within a mixed clip, preventing the loss of critical temporal context.
Shapley Value Guidance
The core of our approach is the use of Shapley values from cooperative game theory. We treat body parts as "players" in a game where the "payout" is the model's classification confidence. By calculating the contribution of each part combination, we estimate the saliency of specific body regions for a given action.
When mixing a majority (head) class with a minority (tail) class, the Tail-Aware Mixing Policy ensures that the most salient parts of the tail action are preserved in the final synthetic sample. For instance, if "waving" is a tail class, the algorithm prioritizes keeping the arm motion from the "wave" sample, ensuring the resulting data is representative rather than confusing.
Experimental Evaluation
We evaluated Shap-Mix on long-tailed variants of the NTU RGB+D 60/120 and the naturally long-tailed Kinetics-Skeleton 400 datasets.
- Performance Gains: On LT-NTU 60, Shap-Mix achieved 80.8% accuracy, a 2.0% improvement over previous state-of-the-art methods.
- Tail Class Improvement: The most significant impact was observed in few-shot categories (<20 samples), where accuracy improved by nearly 12% compared to standard cross-entropy training.
- Robustness: ST-Mix alone proved to be a superior baseline for skeleton augmentation, even on balanced datasets, highlighting the importance of part-level spatial-temporal consistency.
Conclusion
Shap-Mix provides a principled way to handle data imbalance in skeleton-based action recognition. By leveraging game-theoretic saliency, it moves beyond random augmentation to create "smarter" synthetic samples that respect the underlying semantics of human motion. While calculating Shapley values adds some computational overhead, the significant gains in tail-class performance make it a powerful tool for real-world deployment.
Summary updated