Machine Learning-Based Classification of Dependence in Ambulation in Stroke Patients Using Smartphone Video Data

Jong Taek Lee, Eunhee Park, Tae‐Du JungJournal of Personalized Medicine 2021 · DOI

TL;DR

This paper introduces a novel deep learning framework that uses smartphone videos to automatically classify the level of ambulation dependence in stroke patients. By combining a 3D convolutional neural network (3D-CNN) with swing time asymmetry analysis, the system achieves high accuracy, offering a practical tool for monitoring patient recovery and preventing falls.

What Problem Does This Paper Address?

Stroke is a leading cause of acquired disability, significantly impacting a patient's ability to walk independently. While rehabilitation aims to restore this function, a substantial portion of stroke survivors continue to require assistance or are unable to walk. Accurately assessing a patient's level of dependence in ambulation is crucial for designing effective rehabilitation strategies and preventing falls, which are a major concern for this population. Current clinical assessments, like the Functional Ambulatory Categories (FACs) and Berg Balance Scale (BBS), are time-consuming and require trained personnel. Furthermore, privacy concerns can limit the use of clinical assessment data in certain settings. This paper addresses the need for an automated, accessible, and privacy-preserving method to classify ambulation dependence using readily available technology like smartphones.

Methods

The core of the proposed framework is a deep learning model based on a 3D Convolutional Neural Network (3D-CNN), specifically adapted from the MoViNet architecture. To handle the limited dataset size, the researchers employed transfer learning from a pre-trained model.

Here's a breakdown of the key methodological steps:

  • Video Data Collection: Videos of stroke patients undergoing inpatient rehabilitation therapy were recorded using smartphones.
  • Region-of-Interest Extraction: The OpenPose library was used for real-time multi-person pose estimation to identify and track individuals. A clinician then manually labeled the stroke patient of interest. This allowed for the extraction of "patient-centered video" and "patient-centered pose" data, effectively isolating the subject and minimizing background noise.
  • 3D-CNN for Classification: The patient-centered video segments were fed into the adapted MoViNet-A2 3D-CNN model. This model was modified for a 5 fps input and a 5-second clip length to optimize processing time and quality. The network was trained to classify patients as either dependent or independent in their ambulation.
  • Swing Time Asymmetry Analysis: To enhance classification accuracy, particularly in ambiguous cases, the researchers incorporated swing time asymmetry. This metric, derived from the patient's pose keypoints (specifically heel and toe movements), measures the difference in the time it takes for the paretic and non-paretic legs to swing. This feature was used as a supplementary input to the classification model.

Datasets

The study utilized video data collected from 206 stroke patients who received inpatient rehabilitation therapy at Kyungpook National University Chilgok Hospital.

  • Data Source: Smartphone videos recorded during physical therapy sessions.
  • Total Clips: 2311 video clips were initially collected.
  • Walk Action Clips: 1218 clips specifically featuring the "walk" action were selected for analysis, representing approximately 52.7% of the total.
  • Patient Demographics: The patients were aged 63.24 ± 14.36 years, with a mix of ischemic and hemorrhagic stroke types.
  • Ground Truth: Dependence in ambulation was assessed by physiotherapists using the Functional Ambulatory Categories (FACs) and the Berg Balance Scale (BBS). A FAC score < 4 or a BBS score < 45 was defined as dependent ambulation.

No new benchmark datasets were introduced; the study used existing clinical assessment tools for ground truth.

Key Insights and Ideas

  • Smartphone Video as a Viable Data Source: The study demonstrates that readily available smartphone video data, captured in a naturalistic rehabilitation setting, can be effectively used for complex clinical assessments like ambulation dependence classification. This democratizes the technology and makes it more accessible.
  • Synergy of 3D-CNN and Kinematic Features: Combining a powerful deep learning model (3D-CNN) with a biomechanical feature (swing time asymmetry) significantly boosted performance. This highlights the benefit of integrating different types of information for more robust predictions.
  • Addressing Ambiguity with Kinematic Data: The swing time asymmetry analysis proved particularly useful in cases where the 3D-CNN's output was uncertain (around 0.4-0.6). This suggests that kinematic features can act as a crucial "tie-breaker" for improved diagnostic confidence.
  • Privacy-Preserving Potential: By analyzing video data without extracting personal identifying information, the framework offers a privacy-conscious alternative to traditional clinical assessments, which could be critical for adoption in real-world healthcare settings.

Key Results

The study evaluated the performance of the 3D-CNN model using fivefold cross-validation.

  • 3D-CNN (FACs and BBS based): Achieved an accuracy of 86.3%, precision of 87.4%, recall of 94.0%, and an F1 score of 90.5%.
  • 3D-CNN + Swing Time Asymmetry: This combined approach showed improved performance, reaching 88.7% accuracy, 89.1% precision, 95.7% recall, and a 92.2% F1 score.
  • Receiver Operating Characteristic (ROC) Curves: The AUC for dependent ambulation was 0.93 with the 3D-CNN alone and improved to 0.94 when swing time asymmetry was included.

Limitations and Future Work

The authors acknowledge several limitations and suggest avenues for future research:

  • Image Noise: The study did not apply image denoising techniques, which could potentially improve performance if the input images are noisy.
  • Limited Feature Set: Only swing time asymmetry was analyzed. Future work could explore other spatiotemporal parameters like step length and velocity using 3D pose estimation.
  • Video-Only Analysis: The framework currently relies solely on video data. Incorporating audio data from smartphones could provide additional robustness, especially in cases of occlusion.
  • Bounding Box vs. Segmentation: While bounding boxes were used for person detection, image segmentation could offer more precise information about patient posture.

Future directions include estimating various spatiotemporal parameters during walking using 3D pose estimation, exploring the integration of audio data, and investigating image segmentation for more detailed posture analysis.

Summary updated

Abstract

The goal of this study was to develop a framework to classify dependence in ambulation by employing a deep model in a 3D convolutional neural network (3D-CNN) using video data recorded by a smartphone during inpatient rehabilitation therapy in stroke patients. Among 2311 video clips, 1218 walk action cases were collected from 206 stroke patients receiving inpatient rehabilitation therapy (63.24 ± 14.36 years old). As ground truth, the dependence in ambulation was assessed and labeled using the functional ambulatory categories (FACs) and Berg balance scale (BBS). The dependent ambulation was defined as a FAC score less than 4 or a BBS score less than 45. We extracted patient-centered video and patient-centered pose of the target from the tracked target's posture keypoint location information. Then, the extracted patient-centered video was input in the 3D-CNN, and the extracted patient-centered pose was used to measure swing time asymmetry. Finally, we evaluated the classification of dependence in ambulation using video data via fivefold cross-validation. When training the 3D-CNN based on FACs and BBS, the model performed with 86.3% accuracy, 87.4% precision, 94.0% recall, and 90.5% F1 score. When the 3D-CNN based on FACs and BBS was combined with swing time asymmetry, the model exhibited improved performance (88.7% accuracy, 89.1% precision, 95.7% recall, and 92.2% F1 score). The proposed framework for dependence in ambulation can be useful, as it alerts clinicians or caregivers when stroke patients with dependent ambulatory move alone without assistance. In addition, monitoring dependence in ambulation can facilitate the design of individualized rehabilitation strategies for stroke patients with impaired mobility and balance function.
BibTeX
@article{lee2021machine,
  author = {Jong Taek Lee and Eunhee Park and Tae‐Du Jung},
  title = {{Machine Learning-Based Classification of Dependence in Ambulation in Stroke Patients Using Smartphone Video Data}},
  journal = {Journal of Personalized Medicine},
  year = {2021},
  doi = {10.3390/jpm11111080},
}
Figure