A Skeleton-Based Rehabilitation Exercise Assessment System With Rotation Invariance

Kaili Zheng, Ji Wu, Jialin Zhang, Chenyi GuoIEEE Transactions on Neural Systems and Rehabilitation Engineering 2023 · DOI

Abstract

Automated exercise assessment is of great importance for patients under rehabilitation exercise who require professional guidance. Among the existing approaches, the skeleton-based assessment model that classifies the correctness of the exercise has attracted much attention due to its relative ease of implementation and convenience in use. However, there are two problems with this approach: sensitivity to the orientation of the human skeleton and a lack of actionable feedback. To solve these, we propose a novel rotation-invariant descriptor—the dot product matrix of the human skeleton—and a visualization method based on Gradient-Weighted Class Activation Mapping (Grad-CAM). Experimental results on two public datasets and a self-generated push-up dataset show that our method achieves absolute robustness to orientation and provides informative visualization for movement correction.

Introduction

Automated rehabilitation systems are essential for patients recovering at home, providing the guidance typically offered by physical therapists. Skeleton-based AI models are often preferred over image-based models because they are computationally lighter and more privacy-preserving. However, current systems face two major hurdles: orientation sensitivity and the "black box" feedback gap. Most models fail if a patient is not facing the camera directly, and they often only provide a binary "correct/incorrect" label without explaining why a movement was flagged.

Comparison of image-based and skeleton-based assessment pipelines.

Methodology

The proposed system integrates a two-fold solution into a Spatial-Temporal Graph Convolutional Network (ST-GCN).

Rotation-Invariant Descriptor

Instead of using raw XYZ joint coordinates, which vary with camera angle, the system transforms the skeleton into a Dot Product Matrix (DPM). By calculating the dot product of vectors for all joint pairs, the system captures the body's relative geometry while mathematically discarding global orientation. This ensures "absolute robustness" to rotation, allowing the model to focus entirely on the spatial-temporal features of the movement rather than the user's position relative to the sensor.

Visualized Feedback via Grad-CAM

To provide actionable feedback, the authors adapted Gradient-Weighted Class Activation Mapping (Grad-CAM). By backpropagating the classification score, the system identifies which joints contributed most to an "incorrect" label. These joints are highlighted on a 3D skeleton, allowing users to see exactly which part of their form needs adjustment (e.g., "elbows too wide"). The authors also introduced the Overlap Ratio (OvR) to quantitatively measure the quality of these visualizations.

Datasets

The model was rigorously evaluated on three distinct datasets to ensure broad applicability:

  1. IRDS (Intelligent Rehabilitation DataSet): Contains 9 movement types from 29 subjects.
  2. UI-PRMD: Features 10 common rehab exercises captured via both Kinect and high-precision Vicon systems.
  3. Push-Up Dataset: A new, publicly released dataset of 529 samples from 16 subjects, specifically designed to test rotation invariance and common errors like "collapsed waist" or "forward extended neck."
Skeleton data samples from the IRDS and UI-PRMD (Kinect and Vicon) datasets.

Key Results

  • Absolute Robustness: In stress tests with up to 90-degree rotations, baseline models dropped to ~40-60% accuracy, while the proposed DPM method maintained near-perfect accuracy (~98-100%).
  • Superior Performance: On the push-up dataset, the rotation-invariant model achieved 85.44% accuracy, outperforming vanilla ST-GCN and other state-of-the-art normalization techniques.
  • Effective Feedback: Qualitative results confirmed that the system accurately highlights specific errors, providing clear guidance for patients to correct their movements independently.

Conclusion

By introducing a rotation-invariant descriptor and a visualization-based feedback mechanism, this research addresses the primary limitations of skeleton-based exercise assessment. The system not only remains robust across various camera orientations but also bridges the feedback gap, making home-based rehabilitation more effective and accessible.

The code and push-up dataset are available at https://github.com/Kelly510/RehabExerAssess.

Summary updated

Abstract

Automated exercise assessment is of great importance for patients under rehabilitation exercise who require professional guidance. Among the existing approaches, the skeleton-based assessment model that classifies the correctness of the exercise has attracted much attention due to its relative ease of implementation and convenience in use. However, there are two problems with this approach. The first problem is its sensitivity to the orientation of the human skeleton. To solve this problem, we propose a novel rotation-invariant descriptor, the dot product matrix of the human skeleton, and prove mathematically that this descriptor discards only the orientation message that we do not expect while preserving all other useful information. Lack of feedback from the system is the second problem, because the exercisers do not know which parts of their exercises are incorrect. Therefore, we develop a visualization method for our system based on Gradient-Weighted Class Activation Mapping (Grad-CAM) and an quantitative metric called Overlap Ratio (OvR) to measure the quality of the visualization result. To demonstrate the effect of our method, we conduct experiments on two public datasets and a self-generated push-up dataset. The experimental results show that our rotation-invariant descriptor can achieve absolute robustness to orientation even under severe angle perturbations. In terms of accuracy and OvR, our method even outperforms previous works in most cases, indicating that the rotation-invariant descriptor helps the assessment model to extract more stable features. The visualization results are also informative to correct the movements; some examples are presented in this paper. The code of this paper and our push-up dataset are publicly available at https://github.com/Kelly510/RehabExerAssess.
BibTeX
@article{zheng2023skeleton,
  author = {Kaili Zheng and Ji Wu and Jialin Zhang and Chenyi Guo},
  title = {{A Skeleton-Based Rehabilitation Exercise Assessment System With Rotation Invariance}},
  journal = {IEEE Transactions on Neural Systems and Rehabilitation Engineering},
  year = {2023},
  doi = {10.1109/tnsre.2023.3282675},
}
Figure