TL;DR
This paper introduces "Upbeat," an augmented reality (AR) dance game designed to make upper limb prosthetic rehabilitation more engaging and effective. By combining motion tracking, EMG data, and gamified feedback, Upbeat aims to improve patient motivation and provide objective progress assessment for therapists.
What Problem Does This Paper Address?
Millions of people worldwide undergo upper limb amputations each year, and a significant portion struggle with rehabilitation, leading to dissatisfaction and rejection of prosthetic devices. Key challenges in current rehabilitation include:
- Late start of therapy: Patients often have to wait for prosthetic fitting, delaying crucial early intervention and potentially leading to muscle atrophy.
- Lack of comprehensive assessment: Traditional tests are often limited in scope, only measuring completion rates and not providing detailed insights into movement quality or compensatory strategies.
- Poor patient motivation: The repetitive and often tedious nature of rehabilitation exercises can lead to low patient engagement and commitment.
This paper proposes an AR-based solution to address these issues by making rehabilitation more accessible, objective, and enjoyable.
Methods
Upbeat is built on a foundation of AR guidance, gesture recognition, and markerless body tracking. The core components include:
- Virtual Dance Instructor: A 3D avatar guides the patient through choreographed dance routines that incorporate specific hand gestures relevant to upper limb rehabilitation.
- Motion Tracking: A Microsoft Kinect sensor captures the patient's body position and movements in real-time.
- EMG Signal Analysis: A Myo Armband worn on the forearm detects electromyography (EMG) signals from the patient's muscles. This data is used to classify hand gestures.
- AR Visualization (Mirracle): The Mirracle system overlays a visualization of the patient's upper arm muscle activity onto their real-time video feed, showing which muscles are activated (green) or relaxed (red).
- Gamification: A scoring system awards points for correctly performed gestures within the allotted time, providing immediate feedback and encouraging engagement.
- Post-Game Feedback: A diagnostic summary includes a gamified score, an EMG graph illustrating muscle activity patterns, and the AR visualization of muscle activation.
The system was developed using the Unity 3D game engine, integrating the Kinect and Myo Armband SDKs. Choreography was captured using motion capture and animated onto the virtual instructor.
Datasets
The paper does not explicitly mention training datasets. The system was evaluated on three subjects performing 10 trials each. The evaluation focused on the system's ability to correctly classify hand gestures and measure performance metrics like detection, reaction, and operating times.
Key Insights and Ideas
- Gamifying Rehabilitation: Transforming repetitive therapy into a dance game significantly increases patient engagement and motivation, a critical factor for successful rehabilitation.
- Early Intervention with AR: AR allows patients to begin rehabilitation immediately after trauma, even before a prosthetic is fitted, by using a virtual limb and real-time muscle feedback.
- Objective, Personalized Feedback: Combining EMG data, motion tracking, and AR visualization provides therapists with detailed, objective insights into a patient's progress and muscle activation patterns, enabling personalized therapy adjustments.
- Bridging the Gap: Upbeat builds upon existing protocols like PHAM by integrating them into a more accessible and engaging AR framework, aiming to improve upon current limitations.
Key Results
- Gesture Classification Accuracy: The system achieved an average classification accuracy of 77% for the hand gestures used in the study. While slightly lower than the commercial Myo Armband's reported accuracy (82.8%), this is attributed to the integration within the Unity environment and potential calibration challenges.
- Performance Metrics:
- Average detection time: 0.24 ± 0.31 seconds
- Average reaction time: 0.92 ± 0.10 seconds
- Average operating time: 1.15 ± 0.34 seconds
- Timeliness: The system can detect gestures efficiently within an average of 2.62 seconds, which is well within the 6-12 second time intervals of the current dance movements, suggesting suitability for faster-paced choreography in future versions.
- Qualitative Feedback: The system successfully provided visual feedback on muscle activity and a gamified score, offering both quantitative and qualitative insights into performance.
Limitations and Future Work
- Calibration Robustness: Misclassification was more prevalent in one subject, highlighting the need for a more robust calibration protocol to ensure consistent performance across users.
- Gesture Complexity: The current hand gestures have distinct EMG patterns. Expanding to more complex gestures or faster movements might require further refinement of the EMG classification.
- Clinical Validation: The current study is a proof of concept. The next crucial step is to conduct clinical trials with upper limb amputees to evaluate Upbeat's efficacy compared to traditional rehabilitation methods.
- Subjective Assessment: While objective data is strong, a subjective assessment of how enjoyable and engaging the system is for the target user group is also a future research direction.
- Advanced Features: Future work includes implementing more complex hand gestures, introducing different difficulty levels, and refining the scoring system to account for compensatory movements for a more nuanced performance evaluation.
Summary updated