Upbeat: Augmented Reality-Guided Dancing for Prosthetic Rehabilitation of Upper Limb Amputees

Marina Melero, Annie Hou, Emily Cheng, Amogh Tayade, Sing Chun Lee, Mathias Unberath, Nassir NavabJournal of Healthcare Engineering 2019 · DOI

TL;DR

This paper introduces "Upbeat," an augmented reality (AR) dance game designed to make upper limb prosthetic rehabilitation more engaging and effective. By combining motion tracking, EMG data, and gamified feedback, Upbeat aims to improve patient motivation and provide objective progress assessment for therapists.

What Problem Does This Paper Address?

Millions of people worldwide undergo upper limb amputations each year, and a significant portion struggle with rehabilitation, leading to dissatisfaction and rejection of prosthetic devices. Key challenges in current rehabilitation include:

  • Late start of therapy: Patients often have to wait for prosthetic fitting, delaying crucial early intervention and potentially leading to muscle atrophy.
  • Lack of comprehensive assessment: Traditional tests are often limited in scope, only measuring completion rates and not providing detailed insights into movement quality or compensatory strategies.
  • Poor patient motivation: The repetitive and often tedious nature of rehabilitation exercises can lead to low patient engagement and commitment.

This paper proposes an AR-based solution to address these issues by making rehabilitation more accessible, objective, and enjoyable.

Methods

Upbeat is built on a foundation of AR guidance, gesture recognition, and markerless body tracking. The core components include:

  • Virtual Dance Instructor: A 3D avatar guides the patient through choreographed dance routines that incorporate specific hand gestures relevant to upper limb rehabilitation.
  • Motion Tracking: A Microsoft Kinect sensor captures the patient's body position and movements in real-time.
  • EMG Signal Analysis: A Myo Armband worn on the forearm detects electromyography (EMG) signals from the patient's muscles. This data is used to classify hand gestures.
  • AR Visualization (Mirracle): The Mirracle system overlays a visualization of the patient's upper arm muscle activity onto their real-time video feed, showing which muscles are activated (green) or relaxed (red).
  • Gamification: A scoring system awards points for correctly performed gestures within the allotted time, providing immediate feedback and encouraging engagement.
  • Post-Game Feedback: A diagnostic summary includes a gamified score, an EMG graph illustrating muscle activity patterns, and the AR visualization of muscle activation.

The system was developed using the Unity 3D game engine, integrating the Kinect and Myo Armband SDKs. Choreography was captured using motion capture and animated onto the virtual instructor.

Datasets

The paper does not explicitly mention training datasets. The system was evaluated on three subjects performing 10 trials each. The evaluation focused on the system's ability to correctly classify hand gestures and measure performance metrics like detection, reaction, and operating times.

Key Insights and Ideas

  • Gamifying Rehabilitation: Transforming repetitive therapy into a dance game significantly increases patient engagement and motivation, a critical factor for successful rehabilitation.
  • Early Intervention with AR: AR allows patients to begin rehabilitation immediately after trauma, even before a prosthetic is fitted, by using a virtual limb and real-time muscle feedback.
  • Objective, Personalized Feedback: Combining EMG data, motion tracking, and AR visualization provides therapists with detailed, objective insights into a patient's progress and muscle activation patterns, enabling personalized therapy adjustments.
  • Bridging the Gap: Upbeat builds upon existing protocols like PHAM by integrating them into a more accessible and engaging AR framework, aiming to improve upon current limitations.

Key Results

  • Gesture Classification Accuracy: The system achieved an average classification accuracy of 77% for the hand gestures used in the study. While slightly lower than the commercial Myo Armband's reported accuracy (82.8%), this is attributed to the integration within the Unity environment and potential calibration challenges.
  • Performance Metrics:
    • Average detection time: 0.24 ± 0.31 seconds
    • Average reaction time: 0.92 ± 0.10 seconds
    • Average operating time: 1.15 ± 0.34 seconds
  • Timeliness: The system can detect gestures efficiently within an average of 2.62 seconds, which is well within the 6-12 second time intervals of the current dance movements, suggesting suitability for faster-paced choreography in future versions.
  • Qualitative Feedback: The system successfully provided visual feedback on muscle activity and a gamified score, offering both quantitative and qualitative insights into performance.

Limitations and Future Work

  • Calibration Robustness: Misclassification was more prevalent in one subject, highlighting the need for a more robust calibration protocol to ensure consistent performance across users.
  • Gesture Complexity: The current hand gestures have distinct EMG patterns. Expanding to more complex gestures or faster movements might require further refinement of the EMG classification.
  • Clinical Validation: The current study is a proof of concept. The next crucial step is to conduct clinical trials with upper limb amputees to evaluate Upbeat's efficacy compared to traditional rehabilitation methods.
  • Subjective Assessment: While objective data is strong, a subjective assessment of how enjoyable and engaging the system is for the target user group is also a future research direction.
  • Advanced Features: Future work includes implementing more complex hand gestures, introducing different difficulty levels, and refining the scoring system to account for compensatory movements for a more nuanced performance evaluation.

Summary updated

Abstract

Unsuccessful rehabilitation therapy is a widespread issue amongst modern day amputees. Of the estimated 10 million amputees worldwide, 3 million of whom are upper limb amputees, a large majority are discontent and experience rejection with their current prosthesis during activities of daily living (ADL). Here we introduce <i>Upbeat</i>, an augmented reality (AR) dance game designed to improve rehabilitation therapies in upper limb amputees. In <i>Upbeat</i>, the patient is instructed to follow a virtual dance instructor, performing choreographed dance movements containing hand gestures involved in upper limb rehabilitation therapy. The patient's position is then tracked using a Microsoft Kinect sensor while the hand gestures are analyzed using EMG data collected from a Myo Armband. Additionally, a gamified score is calculated based on how many gestures and movements were correctly performed. Upon completion of the game, a diagnostic summary of the results is shown in the form of a graph summarizing the collected EMG data, as well as with a video displaying an augmented visualization of the patient's upper arm muscle activity during gameplay. By gamifying the rehabilitation process, <i>Upbeat</i> has the potential to improve therapy on upper limb amputees by enabling the start of rehabilitation immediately after trauma, providing personalized feedback which professionals can utilize to accurately assess patient's progress, and increasing patient excitement, therefore increasing patient willingness to complete rehabilitation. This paper is concerned with the description and evaluation of our prototypic implementation of <i>Upbeat</i> that will serve as the basis for conducting clinical studies to evaluate its impact on rehabilitation.
BibTeX
@article{melero2019upbeat,
  author = {Marina Melero and Annie Hou and Emily Cheng and Amogh Tayade and Sing Chun Lee and Mathias Unberath and Nassir Navab},
  title = {{Upbeat: Augmented Reality-Guided Dancing for Prosthetic Rehabilitation of Upper Limb Amputees}},
  journal = {Journal of Healthcare Engineering},
  year = {2019},
  doi = {10.1155/2019/2163705},
}
Figure