Detection of Rehabilitation Training Effect of Upper Limb Movement Disorder Based on MPL-CNN

Lijuan Shi, Runmin Wang, Jian Zhao, Jing Zhang, Zhejun KuangSensors 2024 · DOI

TL;DR

This paper introduces MPL-CNN, a novel deep learning model that leverages Mediapipe for real-time upper limb movement analysis to accurately detect the effectiveness of rehabilitation training for stroke patients. By combining spatial and temporal feature extraction, the model achieves high accuracy in recognizing rehabilitation actions, paving the way for more personalized and precise patient care.

What Problem Does This Paper Address?

Stroke is a leading cause of physical disability, often resulting in upper limb movement disorders. Effective rehabilitation is crucial for recovery, but its evaluation can be challenging due to subjective assessments and limited resources. Current methods struggle to provide objective, real-time feedback on the quality and effectiveness of rehabilitation exercises. This paper aims to bridge this gap by developing an automated system that can accurately assess upper limb rehabilitation training progress, enabling clinicians to better understand patient needs and tailor treatment plans.

Methods

The core of this work is the MPL-CNN model, a hybrid deep learning architecture designed for action recognition. It starts by using Mediapipe, specifically its BlazePose and BlazeHands models, to extract keypoint features from both the upper limbs and hands of patients in real-time. This is a key innovation as it captures a richer set of features than traditional pose estimation methods that might focus solely on the body.

These extracted spatial features are then fed into a Long Short-Term Memory (LSTM) network. LSTMs are well-suited for processing sequential data, allowing the model to capture the temporal dynamics and evolution of movements over time. Finally, these temporal features are processed by a Convolutional Neural Network (CNN) to further extract spatial patterns and refine the action recognition. This combination of Mediapipe for feature extraction, LSTM for temporal modeling, and CNN for spatial feature refinement forms the MPL-CNN model.

Datasets

The study utilized a custom dataset created for this research. This dataset consists of videos of upper limb rehabilitation training movements performed by ten neurologically normal adults. The movements were based on the Fugl-Meyer Upper Limb Rehabilitation Training Functional Assessment Form (FMA), a standard clinical assessment tool. The dataset includes various types of movements, such as flexor synergistic movements, common extensor muscle movements, isolation movements, wrist stability exercises, and hand movements. The data was collected at 30 frames per second and is divided into training and testing sets with a 4:1 ratio.

Key Insights and Ideas

  • Unified Hand and Body Pose Estimation: The integration of Mediapipe's BlazePose and BlazeHands is a significant advantage. It allows for simultaneous extraction of both upper limb and hand keypoints, providing a more comprehensive understanding of fine motor control crucial for rehabilitation.
  • Synergy of Temporal and Spatial Features: The MPL-CNN model's strength lies in its ability to effectively combine temporal information (captured by LSTM) with spatial information (extracted by Mediapipe and further processed by CNN). This holistic approach is vital for accurately recognizing complex rehabilitation movements.
  • Real-time and Objective Assessment: The use of Mediapipe enables real-time processing, allowing for immediate feedback on rehabilitation exercises. This moves away from subjective clinical assessments towards objective, data-driven evaluations.
  • Adaptability to Standardized Protocols: By basing the dataset on the FMA, the model is directly applicable to established clinical rehabilitation standards, making its integration into clinical practice more feasible.

Key Results

The MPL-CNN model demonstrated impressive performance in detecting upper limb rehabilitation training actions. The study reported a recognition accuracy of 95% for individual stages of the Fugl-Meyer assessment and an average accuracy of 97.54% across various rehabilitation actions. The model also achieved high recall and F1-scores, generally above 90%, indicating its robustness and reliability across different movement categories. The authors also noted that their model showed a significant improvement in accuracy compared to previous methods.

Limitations and Future Work

The authors acknowledge that their study used a dataset collected from neurologically normal individuals. While this ensures the scientific validity of the FMA movements, future work should focus on collecting data from actual stroke patients to further validate the model's performance in real-world clinical scenarios.

A key direction for future work is the deployment of the MPL-CNN model on mobile devices. This would allow patients to monitor their rehabilitation progress anytime, anywhere, and provide valuable data to clinicians for more informed treatment adjustments. The authors also suggest exploring further optimization of the model for even greater efficiency and accuracy.

Summary updated

Abstract

Stroke represents a medical emergency and can lead to the development of movement disorders such as abnormal muscle tone, limited range of motion, or abnormalities in coordination and balance. In order to help stroke patients recover as soon as possible, rehabilitation training methods employ various movement modes such as ordinary movements and joint reactions to induce active reactions in the limbs and gradually restore normal functions. Rehabilitation effect evaluation can help physicians understand the rehabilitation needs of different patients, determine effective treatment methods and strategies, and improve treatment efficiency. In order to achieve real-time and accuracy of action detection, this article uses Mediapipe's action detection algorithm and proposes a model based on MPL-CNN. Mediapipe can be used to identify key point features of the patient's upper limbs and simultaneously identify key point features of the hand. In order to detect the effect of rehabilitation training for upper limb movement disorders, LSTM and CNN are combined to form a new LSTM-CNN model, which is used to identify the action features of upper limb rehabilitation training extracted by Medipipe. The MPL-CNN model can effectively identify the accuracy of rehabilitation movements during upper limb rehabilitation training for stroke patients. In order to ensure the scientific validity and unified standards of rehabilitation training movements, this article employs the postures in the Fugl-Meyer Upper Limb Rehabilitation Training Functional Assessment Form (FMA) and establishes an FMA upper limb rehabilitation data set for experimental verification. Experimental results show that in each stage of the Fugl-Meyer upper limb rehabilitation training evaluation effect detection, the MPL-CNN-based method's recognition accuracy of upper limb rehabilitation training actions reached 95%. At the same time, the average accuracy rate of various upper limb rehabilitation training actions reaches 97.54%. This shows that the model is highly robust across different action categories and proves that the MPL-CNN model is an effective and feasible solution. This method based on MPL-CNN can provide a high-precision detection method for the evaluation of rehabilitation effects of upper limb movement disorders after stroke, helping clinicians in evaluating the patient's rehabilitation progress and adjusting the rehabilitation plan based on the evaluation results. This will help improve the personalization and precision of rehabilitation treatment and promote patient recovery.
BibTeX
@inproceedings{shi2024detection,
  author = {Lijuan Shi and Runmin Wang and Jian Zhao and Jing Zhang and Zhejun Kuang},
  title = {{Detection of Rehabilitation Training Effect of Upper Limb Movement Disorder Based on MPL-CNN}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24041105},
}
Figure