EGCN: An Ensemble-based Learning Framework for Exploring Effective Skeleton-based Rehabilitation Exercise Assessment

Bruce X. B. Yu, Yan Liu, Xiang Zhang, Gong Chen, Keith C. C. ChanIJCAI 2022 · DOI

TL;DR

This paper introduces EGCN, a novel ensemble-based learning framework that leverages both skeleton position and orientation data to improve the assessment of rehabilitation exercises. By exploring various ensemble strategies, EGCN significantly outperforms existing methods, offering a more robust and accurate way to evaluate exercise quality.

What Problem Does This Paper Address?

Musculoskeletal disorders are a growing global health concern, and physical therapy exercises are crucial for recovery. However, traditional rehabilitation often involves costly in-person sessions. While home-based rehabilitation offers a more affordable alternative, patient adherence and the accurate assessment of exercise quality remain significant challenges. Existing automated systems often rely on single types of skeleton data (either position or orientation) and lack sophisticated evaluation criteria. This paper aims to bridge this gap by developing a more comprehensive framework that effectively utilizes multiple skeleton data streams for accurate rehabilitation exercise assessment.

Methods

The core of EGCN is an Ensemble-based Graph Convolutional Network (EGCN). The framework first employs a Graph Convolutional Network (GCN) to extract features from two distinct skeleton data streams: position and orientation. These GCN submodels are designed to capture both spatial and temporal information within the skeleton data.

The novelty lies in the ensemble strategies explored:

  • Data-Level Ensemble (SLE): Concatenates position and orientation streams before feeding them into a single GCN.
  • Feature-Level Ensemble (FLE): Separately extracts features from each stream and then concatenates them. FLE-2 further encourages feature diversity by penalizing cosine similarity between features.
  • Decision-Level Ensemble (DLE): Aggregates predictions from separate GCNs trained on each stream. DLE-1 trains submodels jointly, while DLE-2 trains them independently.
  • Model-Level Ensemble (MLE): This is the proposed key contribution. It leverages the intuition that position data captures global movement and orientation captures local joint characteristics. MLE uses the neuron activation values from the position stream's GCN (g) to inform and regularize the training of the orientation stream's GCN (h). Specifically, it uses the "joint importance" derived from g to guide the learning of h through element-wise multiplication.

Datasets

The paper evaluates EGCN on two publicly available datasets:

  • UI-PRMD: Contains skeletal data from 10 healthy subjects performing 10 rehabilitation exercises, with both correct and incorrect repetitions. The dataset used is a cleaned version with 1,326 repetitions.
  • KIMORE: Collected from 78 subjects categorized into control and patient groups, featuring 5 exercises. The dataset is manually segmented to label repetitions as normal or abnormal.

Key Insights and Ideas

  • Synergy of Position and Orientation: The paper strongly suggests that combining both position and orientation skeleton features is crucial for effective rehabilitation exercise assessment, as they capture complementary information.
  • Model-Level Ensemble's Power: The proposed Model-Level Ensemble (MLE) strategy, which uses position data to guide orientation learning, consistently outperforms other ensemble strategies and single-modal methods. This highlights the benefit of a more integrated fusion approach rather than simple concatenation or aggregation.
  • Importance of "Fixing" Pretrained Models: Ablation studies reveal that fixing the pretrained parameters of the position GCN (g) in the MLE strategy is the most effective setting, suggesting that the learned "joint importance" from position data is a valuable regularizer for the orientation stream.
  • Limitations of Existing Evaluation Metrics: The paper critically analyzes existing evaluation metrics like Distance Metric (DM) and Separation Degree (SD), finding them insufficient for assessing prediction ability and potentially misleading. They advocate for cross-validation to evaluate prediction capabilities.

Key Results

The EGCN framework, particularly with the Model-Level Ensemble (MLE) strategy, achieved superior performance across both datasets. For instance, on the UI-PRMD dataset, the MLE strategy achieved an average accuracy of 86.9% with the basic GCN backbone, significantly outperforming other ensemble methods and single-modal approaches. Similar improvements were observed on the KIMORE dataset. The paper also demonstrates that using advanced GCN backbones (like AGCN or MS-G3D) does not always lead to stable improvements, suggesting that the proposed ensemble framework is more critical than the specific GCN architecture for this task.

Limitations and Future Work

The authors acknowledge that the UI-PRMD and KIMORE datasets are relatively small. Future work includes investigating if EGCN can generate evaluation scores consistent with human judgment and developing online exercise evaluation methods. They also plan to explore if their EGCN can be generalized to other GCN backbones more effectively.

Summary updated

BibTeX
@inproceedings{yu2022egcn,
  author = {Bruce X. B. Yu and Yan Liu and Xiang Zhang and Gong Chen and Keith C. C. Chan},
  title = {{EGCN: An Ensemble-based Learning Framework for Exploring Effective Skeleton-based Rehabilitation Exercise Assessment}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/511},
}
Figure