Shap-Mix: Shapley Value Guided Mixing for Long-Tailed Skeleton Based Action Recognition

Jiahang Zhang, Lilang Lin, Jiaying Liu

Abstract

Skeleton-based action recognition models often struggle with real-world "long-tailed" data distributions, where common actions are overrepresented and critical rare actions are scarce. Standard augmentation techniques like Mixup often create semantically confusing samples for sparse skeleton data. We introduce Shap-Mix, a data augmentation framework that utilizes Shapley values to identify salient body parts for rare actions. By ensuring these key motion patterns are preserved during the mixing process, Shap-Mix generates representative synthetic samples that effectively regularize the decision boundaries for tail classes. Our method achieves state-of-the-art results on long-tailed versions of NTU RGB+D and Kinetics-Skeleton datasets.

Introduction: The Long-Tail Challenge

In practical applications, human action recognition datasets are rarely balanced. Models typically perform well on "head" classes (e.g., standing, walking) but fail on "tail" classes (e.g., falling, seizure). Skeleton-based models, which represent humans as a series of 3D keypoints, are particularly sensitive to this imbalance. Traditional mixing strategies often lead to "semantic confusion"—synthetic samples that lack physical or logical coherence, thereby degrading the model's ability to learn distinct features for rare actions.

Comparison of Random Mix vs. Shap-Mix and its effect on decision boundaries.

Methodology

The Shap-Mix framework addresses these challenges through a two-stage process: spatial-temporal mixing and Shapley-guided saliency preservation.

Spatial-Temporal Mixing (ST-Mix)

To maintain structural integrity, we move beyond joint-level mixing. ST-Mix operates at the body-part level, dividing the skeleton into five semantic regions (trunk, arms, and legs). Temporally, it employs a down-sampling strategy to preserve the full progression of an action within a mixed clip, preventing the loss of critical temporal context.

Shapley Value Guidance

The core of our approach is the use of Shapley values from cooperative game theory. We treat body parts as "players" in a game where the "payout" is the model's classification confidence. By calculating the contribution of each part combination, we estimate the saliency of specific body regions for a given action.

The Tail-Aware Mixing Policy uses saliency guidance to preserve key motion parts from the minority class.

When mixing a majority (head) class with a minority (tail) class, the Tail-Aware Mixing Policy ensures that the most salient parts of the tail action are preserved in the final synthetic sample. For instance, if "waving" is a tail class, the algorithm prioritizes keeping the arm motion from the "wave" sample, ensuring the resulting data is representative rather than confusing.

Experimental Evaluation

We evaluated Shap-Mix on long-tailed variants of the NTU RGB+D 60/120 and the naturally long-tailed Kinetics-Skeleton 400 datasets.

  • Performance Gains: On LT-NTU 60, Shap-Mix achieved 80.8% accuracy, a 2.0% improvement over previous state-of-the-art methods.
  • Tail Class Improvement: The most significant impact was observed in few-shot categories (<20 samples), where accuracy improved by nearly 12% compared to standard cross-entropy training.
  • Robustness: ST-Mix alone proved to be a superior baseline for skeleton augmentation, even on balanced datasets, highlighting the importance of part-level spatial-temporal consistency.

Conclusion

Shap-Mix provides a principled way to handle data imbalance in skeleton-based action recognition. By leveraging game-theoretic saliency, it moves beyond random augmentation to create "smarter" synthetic samples that respect the underlying semantics of human motion. While calculating Shapley values adds some computational overhead, the significant gains in tail-class performance make it a powerful tool for real-world deployment.

Summary updated

BibTeX
@inproceedings{zhang2024shap,
  author = {Jiahang Zhang and Lilang Lin and Jiaying Liu},
  title = {{Shap-Mix: Shapley Value Guided Mixing for Long-Tailed Skeleton Based Action Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=ahdyTWA69K},
}
Figure