DMMG: Dual Min-Max Games for Self-Supervised Skeleton-Based Action Recognition

Shannan Guan, Xin Yu, Wei Huang, Gengfa Fang, Haiyan Lu

Abstract

In this work, we propose a new Dual Min-Max Games (DMMG) based self-supervised skeleton action recognition method by augmenting unlabeled data in a contrastive learning framework. Our DMMG consists of a viewpoint variation min-max game and an edge perturbation min-max game. These two min-max games adopt an adversarial paradigm to perform data augmentation on the skeleton sequences and graph-structured body joints, respectively. Our viewpoint variation min-max game focuses on constructing various hard contrastive pairs by generating skeleton sequences from various viewpoints. These hard contrastive pairs help our model learn representative action features, thus facilitating model transfer to downstream tasks. Moreover, our edge perturbation min-max game specializes in building diverse hard contrastive samples through perturbing connectivity strength among graph-based body joints. The connectivity-strength varying contrastive pairs enable the model to capture minimal sufficient information of different actions, such as representative gestures for an action while preventing the model from overfitting. By fully exploiting the proposed DMMG, we can generate sufficient challenging contrastive pairs and thus achieve discriminative action feature representations from unlabeled skeleton data in a self-supervised manner. Extensive experiments demonstrate that our method achieves superior results under various evaluation protocols on widely-used NTU-RGB+D and NTU120-RGB+D datasets.

Introduction

Skeleton-based action recognition leverages 3D joint coordinates to identify human activities, offering robustness against environmental changes. However, the reliance on large-scale labeled datasets remains a bottleneck. Self-supervised learning (SSL) via contrastive learning has emerged as a solution, yet standard approaches often rely on random data augmentations. These random methods frequently produce "easy" contrastive pairs that fail to challenge the model, or "misleading" pairs that alter the semantic meaning of an action—for instance, cropping a 'sit-down' action into a 'stand' sequence. Furthermore, models may overfit to non-essential habitual motions rather than core discriminative features.

To address these issues, we propose the Dual Min-Max Games (DMMG) framework. DMMG replaces passive random augmentations with an active adversarial paradigm, forcing the model to learn more discriminative and viewpoint-invariant features.

Comparison between general SSL and DMMG, showing improved feature clustering via t-SNE.

Methodology

The DMMG framework consists of two primary adversarial modules designed to "attack" the encoder, compelling it to capture essential action features.

High-level overview of the (a) Viewpoint Variation and (b) Edge Perturbation Min-Max Games.

Viewpoint Variation Min-Max Game

This game focuses on achieving viewpoint invariance. An augmenter network (the "Min" player) learns to generate quaternions that rotate skeleton sequences into challenging viewpoints, minimizing mutual information between the original and augmented views. Simultaneously, the encoder (the "Max" player) strives to recognize the action despite these extreme rotations, thereby learning representative features that are invariant to camera placement.

Edge Perturbation Min-Max Game

This module targets the structural dependencies of the skeleton. The augmenter learns to weaken or "perturb" the connectivity strength between joints in the graph. By hiding critical joint connections, it forces the encoder to identify actions using only the "minimal sufficient" information. This prevents the model from overfitting to secondary, non-discriminative motions like arm swinging during walking.

Experiments and Results

We evaluated DMMG on the NTU RGB+D 60 and 120 datasets. Our method consistently outperformed state-of-the-art (SOTA) self-supervised methods across various protocols.

  • Linear Evaluation: On NTU-60 Cross-Subject (Xsub), DMMG achieved 82.1% accuracy, a 2.5% improvement over previous benchmarks.
  • Semi-Supervised Learning: With only 1% of labels, DMMG reached 56.1% accuracy, demonstrating superior data efficiency compared to methods hovering around 51-54%.
  • Transfer Learning: Pre-training on NTU-120 and testing on NTU-60 showed strong generalization to unseen action categories, proving that the features learned via DMMG are highly transferable.

Qualitative analysis via t-SNE visualizations confirms that DMMG produces significantly tighter and more distinct clusters for different action classes compared to standard SSL methods, as seen in the comparison between general approaches and our proposed method.

Conclusion

DMMG introduces a novel adversarial approach to self-supervised skeleton-based action recognition. By intelligently generating hard contrastive pairs through viewpoint variation and edge perturbation, the framework enables the learning of highly discriminative and robust action representations. This method sets a new state-of-the-art for self-supervised skeleton action recognition on major benchmarks, offering a more effective alternative to traditional random augmentation strategies.

Summary updated

Abstract

In this work, we propose a new Dual Min-Max Games (DMMG) based self-supervised skeleton action recognition method by augmenting unlabeled data in a contrastive learning framework. Our DMMG consists of a viewpoint variation min-max game and an edge perturbation min-max game. These two min-max games adopt an adversarial paradigm to perform data augmentation on the skeleton sequences and graph-structured body joints, respectively. Our viewpoint variation min-max game focuses on constructing various hard contrastive pairs by generating skeleton sequences from various viewpoints. These hard contrastive pairs help our model learn representative action features, thus facilitating model transfer to downstream tasks. Moreover, our edge perturbation min-max game specializes in building diverse hard contrastive samples through perturbing connectivity strength among graph-based body joints. The connectivity-strength varying contrastive pairs enable the model to capture minimal sufficient information of different actions, such as representative gestures for an action while preventing the model from overfitting. By fully exploiting the proposed DMMG, we can generate sufficient challenging contrastive pairs and thus achieve discriminative action feature representations from unlabeled skeleton data in a self-supervised manner. Extensive experiments demonstrate that our method achieves superior results under various evaluation protocols on widely-used NTU-RGB+D and NTU120-RGB+D datasets.
BibTeX
@article{guan2024dmmg,
  author = {Shannan Guan and Xin Yu and Wei Huang and Gengfa Fang and Haiyan Lu},
  title = {{DMMG: Dual Min-Max Games for Self-Supervised Skeleton-Based Action Recognition}},
  journal = {IEEE Trans. Image Process.},
  year = {2024},
  doi = {10.1109/tip.2023.3338410},
}
Figure