PYSKL: Towards Good Practices for Skeleton Action Recognition

Haodong Duan, Jiaqi Wang, Kai Chen, Dahua Lin

Introduction

Skeleton-based action recognition identifies human activities by tracking joint coordinates, offering a computationally efficient and lighting-robust alternative to raw pixel analysis. However, the field has historically lacked standardization. Variations in pre-processing, data augmentation, and training schedules—often referred to as "hidden" settings—make it difficult to discern whether performance gains stem from architectural innovation or superior training recipes.

PYSKL is a new open-source PyTorch toolbox designed to address this "Wild West" environment. By unifying various algorithms under a single framework, PYSKL reveals that standardized "good practices" in data processing and training can often outweigh complex architectural designs, allowing a simplified model like ST-GCN++ to outperform state-of-the-art methods.

The PYSKL Toolbox and ST-GCN++

PYSKL supports both Graph Convolutional Networks (GCNs) and CNN-based approaches. Central to the toolbox is ST-GCN++, a refined version of the seminal Spatial-Temporal Graph Convolutional Network. While recent trends favor heavy attention mechanisms, ST-GCN++ prioritizes structural simplicity and optimized training:

  • Spatial Module: Incorporates a residual link and utilizes iterative gradient descent to update joint connections dynamically.
  • Temporal Module: Replaces large, single-branch convolutions with a multi-branch "Temporal ConvNet" (TCN) using varying dilation rates to capture multi-scale temporal dynamics.
  • Standardized Training: PYSKL implements uniform sampling (randomly selecting frames across a video) and cosine annealing for learning rates, ensuring all models benefit from modern optimization techniques.

Benchmarking Across Diverse Datasets

PYSKL supports nine major benchmarks, spanning both 3D sensor-based skeletons and 2D skeletons estimated from RGB video:

  • 3D Benchmarks: NTURGB+D and NTURGB+D 120.
  • 2D Benchmarks: Kinetics-400, UCF101, and HMDB51 (extracted via pose estimators).
  • Fine-Grained Benchmarks: FineGYM and Diving48, which focus on complex athletic movements.

Key Insights: The Power of Good Practices

The authors' experiments yield a striking conclusion: applying modern training tricks to the original 2018 ST-GCN model nearly eliminates the performance gap between it and 2021 state-of-the-art models like CTR-GCN. On several benchmarks, the difference shrinks to less than 0.5%.

Furthermore, data augmentation proves critical. Spatial augmentations, such as random rotation for 3D data and random scaling for 2D data, are essential for preventing overfitting to specific camera perspectives. ST-GCN++ demonstrates that optimizing these basic building blocks can yield a model that is both faster and more accurate than its more complex predecessors.

Performance and Efficiency Results

PYSKL achieves state-of-the-art (SOTA) performance on 8 out of 9 tested benchmarks. As shown in the comparison below, ST-GCN++ not only matches or exceeds the accuracy of complex models like CTR-GCN but does so with significantly higher computational efficiency.

Comparison of ST-GCN++ with other models on NTURGB+D, highlighting its superior computational efficiency in GFLOPs and parameter count.

On the NTURGB+D XSub benchmark, ST-GCN++ reached 92.6% accuracy, surpassing the official 92.4% of CTR-GCN. Crucially, ST-GCN++ is leaner, utilizing only 1.39M parameters and 2.80 GFLOPs, compared to the original ST-GCN’s 3.08M parameters and 5.34 GFLOPs. The broad impact of the PYSKL framework is evident across a wide array of datasets, consistently outperforming previous SOTA results, particularly on datasets with 2D skeleton annotations.

PYSKL performance compared to previous SOTA across multiple 3D and 2D skeleton datasets.

Limitations and Future Outlook

While PYSKL provides a robust benchmarking environment, 3D-CNN approaches (like PoseC3D) remain computationally intensive compared to GCNs, despite their high accuracy on 2D data. The current focus is primarily on NTURGB+D benchmarks, but the authors plan to expand the toolbox with more diverse algorithms and newer benchmarks. PYSKL is intended to be an actively maintained repository, evolving alongside the "good practices" of the skeleton action recognition field.

Summary updated

Abstract

We present PYSKL: an open-source toolbox for skeleton-based action recognition based on PyTorch. The toolbox supports a wide variety of skeleton action recognition algorithms, including approaches based on GCN and CNN. In contrast to existing open-source skeleton action recognition projects that include only one or two algorithms, PYSKL implements six different algorithms under a unified framework with both the latest and original good practices to ease the comparison of efficacy and efficiency. We also provide an original GCN-based skeleton action recognition model named ST-GCN++, which achieves competitive recognition performance without any complicated attention schemes, serving as a strong baseline. Meanwhile, PYSKL supports the training and testing of nine skeleton-based action recognition benchmarks and achieves state-of-the-art recognition performance on eight of them. To facilitate future research on skeleton action recognition, we also provide a large number of trained models and detailed benchmark results to give some insights. PYSKL is released at https://github.com/kennymckormick/pyskl and is actively maintained. We will update this report when we add new features or benchmarks. The current version corresponds to PYSKL v0.2.
BibTeX
@inproceedings{duan2022pyskl,
  author = {Haodong Duan and Jiaqi Wang and Kai Chen and Dahua Lin},
  title = {{PYSKL: Towards Good Practices for Skeleton Action Recognition}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3548546},
}
Figure