An Image-Based Interactive Training Method of an Upper Limb Rehabilitation Robot

Changlong Ye, Zun Wang, Suyang Yu, Chunying JiangMachines 2024 · DOI

TL;DR

This paper introduces a novel image-based interactive control method for an upper limb rehabilitation robot. By using a camera to track a patient's healthy limb movements and a human skeleton model, the system generates real-time control signals for the robot to assist the affected limb, aiming for more adaptable and robust rehabilitation.

What Problem Does This Paper Address?

Rehabilitation after conditions like stroke often involves significant upper limb motor dysfunction. While robot-assisted therapy shows promise, current systems face challenges in human-machine interaction, adaptability to individual patient needs, and robustness against environmental variations. Many existing solutions are complex, costly, or require extensive setup, hindering widespread adoption. This paper tackles the need for a more intuitive, adaptable, and user-friendly interactive control system for upper limb rehabilitation robots.

Methods

The core of this work is an image-based interactive control method. Here's a breakdown of the key techniques:

  • Human Skeleton Model and Key Point Extraction: A monocular camera captures the patient's movements. The system utilizes a human skeleton model (specifically mentioning Mediapipe) to detect and track key human body points in real-time from the video feed.
  • 3D Coordinate Estimation: Using the extracted 2D key points and spatial geometry principles, the system estimates the 3D coordinates of the upper limb joints. This involves establishing a local coordinate system and using trigonometric calculations based on measured limb lengths (like AB and BC) and projected image data.
  • Motion Data Analysis and Smoothing: To handle potential limb shaking in patients, the extracted motion data (joint velocity and angular velocity) is smoothed using a local weighted regression (LOESS) algorithm with a cubic weight function. This function is chosen for its robustness against outliers.
  • PID Feedback Control: A Proportional-Integral-Derivative (PID) controller is employed to ensure the robot accurately follows the desired motion trajectory. It establishes a feedback loop by comparing the real-time motion data of the user's healthy wrist with the robot's current state, adjusting control signals accordingly.
  • Mirror Therapy Principle: The system is designed to facilitate "mirror movements," where the robot assists the affected limb to perform actions mirroring the healthy limb's movements, leveraging the principles of mirror therapy for neuroplasticity.

Datasets

The paper does not explicitly mention or use pre-existing datasets for training. Instead, it relies on real-time data captured during experiments. The "dataset" for evaluation consists of time-series data of joint angles collected from experiments involving three types of movements (shoulder joint horizontal swing, elbow joint horizontal swing, and joint collaborative movement), with each movement tested 10 times.

Key Insights and Ideas

  • Leveraging the Healthy Limb: Using the patient's healthy limb as the input for control is a clever way to bypass issues with the affected limb's potentially limited or erratic movements. This enhances adaptability.
  • Robustness through Image Analysis: The reliance on image analysis and a skeleton model, rather than more sensitive biological signals or complex depth sensors, aims to improve robustness against environmental changes and individual differences.
  • Image-Based 3D Reconstruction: The method for estimating 3D joint coordinates from a single monocular camera using spatial geometry and known limb lengths is a practical approach for reducing hardware complexity.
  • Mirror Therapy Integration: The direct application of mirror therapy principles through robot-assisted mirroring offers a promising avenue for stimulating neural recovery.

Key Results

The experimental results demonstrate the effectiveness of the proposed method:

  • High Accuracy: The system achieved high accuracy in assisting patients to complete actions, with the robot's movements showing high similarity to the original movements.
  • Real-time Control: The method successfully enables real-time control of the rehabilitation robot. The system responds to experimenter actions within 0.8 seconds.
  • Similarity Metrics: The Dynamic Time Adjustment (DTW) algorithm was used to quantify similarity. Average similarity scores for shoulder and elbow joints were reported to be around 87.95% and 87.70% respectively for horizontal push-pull movements, and 88.18% for push-pull operations.

Limitations and Future Work

The authors acknowledge several limitations:

  • Hand Movement Extraction: The current method struggles to accurately extract motion data for the hand and finger joints due to occlusion, overlap, and the small changes in image length. The method is currently best suited for torso movements.
  • Manual Measurement Errors: The initial measurement of patient limb lengths is done manually, which can introduce errors.
  • Device Specificity: The control method is currently tailored to their specific upper limb rehabilitation robot and would require modification for other devices.

Future work suggestions include:

  • Expanding to Other Robots: Adapting the method for lower limb rehabilitation robots and other types of assistive devices.
  • Improving Recognition Precision: Enhancing the accuracy of recognition and motion data extraction to potentially include hand rehabilitation.
  • Industrial Applications: Exploring the application of posture recognition and control for industrial equipment like robotic arms.

Summary updated

Abstract

Aimed at the problem of human–machine interaction between patients and robots in the process of using rehabilitation robots for rehabilitation training, this paper proposes a human–machine interactive control method based on an independently developed upper limb rehabilitation robot. In this method, the camera is used as a sensor, the human skeleton model is used to analyse the moving image, and the key points of the human body are extracted. Then, the three-dimensional coordinates of the key points of the human arm are extracted by depth estimation and spatial geometry, and then the real-time motion data are obtained, and the control instructions of the robot are generated from it to realise the real-time interactive control of the robot. This method can not only improve the adaptability of the system to individual patient differences, but also improve the robustness of the system, which is less affected by environmental changes. The experimental results show that this method can realise real-time control of the rehabilitation robot, and that the robot assists the patient to complete the action with high accuracy. The results show that this control method is effective and can be applied to the fields of robot control and robot-assisted rehabilitation training.
BibTeX
@inproceedings{ye2024image,
  author = {Changlong Ye and Zun Wang and Suyang Yu and Chunying Jiang},
  title = {{An Image-Based Interactive Training Method of an Upper Limb Rehabilitation Robot}},
  booktitle = {Machines},
  year = {2024},
  doi = {10.3390/machines12050348},
}
Figure