Automated movement assessment in stroke rehabilitation

Tamim Ahmed, Kowshik Thopalli, Thanassis Rikakis, Pavan Turaga, Aisling Kelliher, Jia‐Bin Huang, Steve L. Wolf

TL;DR

This paper introduces the SARAH system, a novel cyber-human approach for automated assessment of upper extremity stroke rehabilitation at home. By fusing expert knowledge with data-driven techniques in a hierarchical model, SARAH achieves robust movement segmentation and task completion scoring from low-cost video data, paving the way for more accessible and effective remote rehabilitation.

What Problem Does This Paper Address?

Stroke rehabilitation is crucial for recovery, but intensive, personalized therapy is often inaccessible due to cost, time, and the need for frequent clinic visits. While telemedicine offers a solution, automating the observation, assessment, and adaptation of physical rehabilitation at home remains a significant challenge. Expert therapists rely on tacit knowledge and nuanced observations that are difficult to quantify and translate into algorithms. Existing low-cost sensing methods like video capture introduce noise and variability, making accurate assessment difficult. This paper aims to bridge this gap by developing an automated system that can reliably assess patient progress during home-based rehabilitation using readily available video data, mimicking the expertise of a therapist.

Methods

The core of SARAH is a hierarchical cyber-human model that integrates expert knowledge with data-driven approaches across different layers of analysis.

  • Lower Layers (Data-Driven):
    • Raw Visual Features: A Multi-Stage Temporal Convolutional Network (MS-TCN) is used to process raw RGB images.
  • Raw Kinematics: A Transformer-based architecture analyzes low-level action features, including tracking of individual body joints (using OpenPose) and object locations (using Faster-RCNN and Kalman filtering).
  • Middle Layer (Integration):
    • Hidden Markov Model (HMM): Leverages kinematic features and incorporates expert-defined transition matrices to model segment sequences. This layer bridges the gap between low-level data and higher-level understanding.
  • Rule-Based Binary Decision Tree (RBBDT): Encodes expert heuristics about key events and relationships (patient-object, patient-space, object-space) to identify segment block transitions and correct remaining errors.
  • Higher Layers (Expert Knowledge-Biased):
    • Ensemble Model: Combines the outputs of the HMM, Transformer, and MS-TCN++ using a weighted average to produce robust per-frame segment labels.
  • Task Completion Assessment: A simple algorithm compares the predicted segment blocks (order, continuity, duration) against expected sequences defined by therapists to assess task completion.

The system also incorporates data denoising and normalization techniques (e.g., z-score based outlier detection with spline interpolation, Mat-Norm) to handle the inherent variability in home-based video data.

Datasets

The study utilized a dataset of 610 captured videos from nine stroke survivors performing the first 12 SARAH tasks. After exclusions, 450 videos of the first ten tasks were used for analysis. Participants repeated each task multiple times. The data was split into training and testing sets using a random split method that ensured no patient-task pair was repeated in both sets.

Key Insights and Ideas

  • Hierarchy is Key: A hierarchical approach that combines expert knowledge (higher layers) with data-driven techniques (lower layers) is crucial for robustly assessing complex human movement in noisy environments.
  • Fusion for Robustness: Combining multiple algorithms (HMM, Transformer, MS-TCN++) through an ensemble approach significantly improves performance by leveraging their complementary strengths. Data-driven models excel at low-level feature extraction, while expert-informed models capture higher-level structure.
  • Expert Knowledge as Priors: Expert-defined segment vocabularies and transition matrices (used in HMM and RBBDT) are vital for guiding the algorithms and overcoming the limitations of noisy, limited data.
  • Home-Based Rehabilitation is Feasible: The SARAH system demonstrates that low-cost, unobtrusive video-based assessment is achievable for long-term, semi-automated rehabilitation at home, even with variable data quality.

Key Results

The SARAH system achieved impressive results:

  • Per-frame labeling accuracy: 85%
  • Segment classification accuracy: 99%
  • Task completion assessment accuracy: 93%

Specifically, the ensemble model combining HMM, Transformer, and MS-TCN++ achieved the best per-frame segmentation performance. The RBBDT algorithm was critical for achieving high accuracy in segment block identification and task completion assessment, with over 99% accuracy in segment block labeling. The system demonstrated robustness to noisy and variable data, a characteristic of home-based rehabilitation.

Limitations and Future Work

  • Data Limitations: The initial dataset was limited, which can affect the generalizability of the models. The authors acknowledge that more data will lead to improved performance.
  • Movement Quality: While the system assesses task completion, it currently does not fully capture nuances of movement quality. Future work aims to integrate therapist ratings to enable automatic assessment of movement quality and its relation to functionality.
  • Generalizability: While the methodology is presented for upper extremity rehabilitation, the authors suggest it can be adapted for other movement rehabilitation contexts (e.g., lower extremity training) and other complex human activities.
  • Real-time Performance: The paper focuses on assessment rather than real-time feedback during the exercise. Future work may explore real-time feedback mechanisms.
  • Pilot Study: The team plans to conduct a pilot study in the homes of stroke survivors to further validate the system.

Summary updated

Abstract

ABSTRACT We are developing a system for long term Semi-Automated Rehabilitation At the Home (SARAH) that relies on low-cost and unobtrusive video-based sensing. We present a cyber-human methodology used by the SARAH system for automated assessment of upper extremity stroke rehabilitation at the home. We propose a hierarchical model for automatically segmenting stroke survivor’s movements and generating training task performance assessment scores during rehabilitation. The hierarchical model fuses expert therapist knowledge-based approaches with data-driven techniques. The expert knowledge is more observable in the higher layers of the hierarchy (task and segment) and therefore more accessible to algorithms incorporating high level constraints relating to activity structure (i.e. type and order of segments per task). We utilize an HMM and a Decision Tree model to connect these high level priors to data driven analysis. The lower layers (RGB images and raw kinematics) need to be addressed primarily through data driven techniques. We use a transformer based architecture operating on low-level action features (tracking of individual body joints and objects) and a Multi-Stage Temporal Convolutional Network(MS-TCN) operating on raw RGB images. We develop a sequence combining these complimentary algorithms effectively, thus encoding the information from different layers of the movement hierarchy. Through this combination, we produce a robust segmentation and task assessment results on noisy, variable and limited data, which is characteristic of low cost video capture of rehabilitation at the home. Our proposed approach achieves 85% accuracy in per-frame labeling, 99% accuracy in segment classification and 93% accuracy in task completion assessment. Although the methodology proposed in this paper applies to upper extremity rehabilitation using the SARAH system, it can potentially be used, with minor alterations, to assist automation in many other movement rehabilitation contexts (i.e. lower extremity training for neurological accidents).
BibTeX
@inproceedings{ahmed2021automated,
  author = {Tamim Ahmed and Kowshik Thopalli and Thanassis Rikakis and Pavan Turaga and Aisling Kelliher and Jia‐Bin Huang and Steve L. Wolf},
  title = {{Automated movement assessment in stroke rehabilitation}},
  booktitle = {Frontiers in Neurology},
  year = {2021},
  doi = {10.3389/fneur.2021.720650},
}
Figure