Automated Movement Assessment in Stroke Rehabilitation

Tamim Ahmed, Kowshik Thopalli, Thanassis Rikakis, Pavan Turaga, Aisling Kelliher, Jia‐Bin Huang, Steven L. Wolf

Abstract We are developing a system for long-term Semi-Automated Rehabilitation At the Home (SARAH) that relies on low-cost and unobtrusive video-based sensing. We present a cyber-human methodology used by the SARAH system for automated assessment of upper extremity stroke rehabilitation at the home. We propose a hierarchical model for automatically segmenting stroke survivor's movements and generating training task performance assessment scores during rehabilitation. The hierarchical model fuses expert therapist knowledge-based approaches with data-driven techniques. The expert knowledge is more observable in the higher layers of the hierarchy (task and segment) and therefore more accessible to algorithms incorporating high-level constraints relating to activity structure. We utilize an HMM and a Decision Tree model to connect these high-level priors to data-driven analysis. The lower layers (RGB images and raw kinematics) are addressed primarily through data-driven techniques, including a transformer-based architecture and a Multi-Stage Temporal Convolutional Network (MS-TCN). Our proposed approach achieves 85% accuracy in per-frame labeling, 99% accuracy in segment classification, and 93% accuracy in task completion assessment.

Introduction: The Challenge of Home-Based Stroke Rehabilitation

Effective stroke recovery requires intensive, long-term therapy. While clinical oversight is the gold standard, the logistical and financial burdens of frequent visits often limit patient access to necessary care. Home-based "tele-rehabilitation" offers a promising alternative, yet it faces a significant technical hurdle: the need for automated, accurate movement assessment.

Current AI systems often struggle to interpret movement with the nuance of a trained therapist. Home environments introduce "noise"—variable lighting, cluttered backgrounds, and awkward camera angles. Furthermore, standard computer vision models frequently lack the context of why a movement is occurring, leading to errors in segmenting complex functional tasks, such as distinguishing between reaching for a cup and actually transporting it. This research introduces a "cyber-human" approach to bridge the gap between raw video data and high-level clinical judgment.

The SARAH System and Methodology

The Semi-Automated Rehabilitation At the Home (SARAH) system is designed to provide high-quality assessment using low-cost, unobtrusive hardware. The setup typically involves a single consumer-grade camera and a specialized activity mat designed to standardize the workspace for functional tasks.

The SARAH system setup (A) and the specialized activity mat (B) used to guide functional tasks.

The core innovation is a five-layer movement hierarchy that connects low-level signals to high-level task goals. This model fuses data-driven techniques with expert therapist logic:

  1. Data-Driven Foundations: At the lower levels, the system employs OpenPose to extract skeletal keypoints and Faster R-CNN to track objects. These signals feed into two architectures: MS-TCN++ (Multi-Stage Temporal Convolutional Network) for recognizing patterns in raw RGB video, and Transformers for modeling long-term dependencies in skeletal movement.
  2. Expert-Driven Logic: To ensure clinical relevance and prevent "illogical" AI predictions, the researchers encoded therapist expertise into the higher layers. Hidden Markov Models (HMMs) enforce the "grammar" of therapy (e.g., a patient must "reach" before they can "transport"), while Rule-Based Binary Decision Trees (RBBDT) identify key transition events based on spatial relationships between the hand and objects.

Results and Clinical Insights

The study utilized a dataset of 450 videos featuring nine stroke survivors with varying levels of impairment performing ten functional tasks. Professional therapists provided the "ground truth" ratings for validation.

The ensemble approach, which combines the data-driven "eyes" of the AI with the "brain" of clinical logic, significantly outperformed individual models. Key performance metrics include:

  • Per-frame labeling accuracy: 85%
  • Segment classification accuracy: 99%
  • Task completion assessment: 93%

A critical finding from the ablation studies was the "Wrist is Enough" discovery: using just the wrist keypoint and object locations often outperformed using the full skeletal pose. This suggests that for functional task assessment, the "end-effector" (the hand) carries the most critical information for clinical evaluation.

Future Directions

While the SARAH system demonstrates robust performance in "noisy" home environments, future work aims to expand its capabilities. Researchers are looking to increase the participant pool to ensure generalization across different stroke pathologies and automate the manual threshold tuning currently required for the decision tree layer. Additionally, the integration of wearable IMU sensors is being explored to provide 3D movement data even when the camera's view is obstructed. The ultimate goal is to transition from post-session analysis to providing real-time audio and visual feedback to patients during their exercises, further empowering independent recovery.

Summary updated

Abstract

We are developing a system for long term Semi-Automated Rehabilitation At the Home (SARAH) that relies on low-cost and unobtrusive video-based sensing. We present a cyber-human methodology used by the SARAH system for automated assessment of upper extremity stroke rehabilitation at the home. We propose a hierarchical model for automatically segmenting stroke survivor's movements and generating training task performance assessment scores during rehabilitation. The hierarchical model fuses expert therapist knowledge-based approaches with data-driven techniques. The expert knowledge is more observable in the higher layers of the hierarchy (task and segment) and therefore more accessible to algorithms incorporating high level constraints relating to activity structure (i.e., type and order of segments per task). We utilize an HMM and a Decision Tree model to connect these high level priors to data driven analysis. The lower layers (RGB images and raw kinematics) need to be addressed primarily through data driven techniques. We use a transformer based architecture operating on low-level action features (tracking of individual body joints and objects) and a Multi-Stage Temporal Convolutional Network(MS-TCN) operating on raw RGB images. We develop a sequence combining these complimentary algorithms effectively, thus encoding the information from different layers of the movement hierarchy. Through this combination, we produce a robust segmentation and task assessment results on noisy, variable and limited data, which is characteristic of low cost video capture of rehabilitation at the home. Our proposed approach achieves 85% accuracy in per-frame labeling, 99% accuracy in segment classification and 93% accuracy in task completion assessment. Although the methodology proposed in this paper applies to upper extremity rehabilitation using the SARAH system, it can potentially be used, with minor alterations, to assist automation in many other movement rehabilitation contexts (i.e., lower extremity training for neurological accidents).
BibTeX
@inproceedings{ahmed2021automated,
  author = {Tamim Ahmed and Kowshik Thopalli and Thanassis Rikakis and Pavan Turaga and Aisling Kelliher and Jia‐Bin Huang and Steven L. Wolf},
  title = {{Automated Movement Assessment in Stroke Rehabilitation}},
  booktitle = {Frontiers in Neurology},
  year = {2021},
  doi = {10.3389/fneur.2021.720650},
}

Loading related content...

Figure