TL;DR
This paper introduces a novel deep learning framework that uses smartphone videos to automatically classify the level of ambulation dependence in stroke patients. By combining a 3D convolutional neural network (3D-CNN) with swing time asymmetry analysis, the system achieves high accuracy, offering a practical tool for monitoring patient recovery and preventing falls.
What Problem Does This Paper Address?
Stroke is a leading cause of acquired disability, significantly impacting a patient's ability to walk independently. While rehabilitation aims to restore this function, a substantial portion of stroke survivors continue to require assistance or are unable to walk. Accurately assessing a patient's level of dependence in ambulation is crucial for designing effective rehabilitation strategies and preventing falls, which are a major concern for this population. Current clinical assessments, like the Functional Ambulatory Categories (FACs) and Berg Balance Scale (BBS), are time-consuming and require trained personnel. Furthermore, privacy concerns can limit the use of clinical assessment data in certain settings. This paper addresses the need for an automated, accessible, and privacy-preserving method to classify ambulation dependence using readily available technology like smartphones.
Methods
The core of the proposed framework is a deep learning model based on a 3D Convolutional Neural Network (3D-CNN), specifically adapted from the MoViNet architecture. To handle the limited dataset size, the researchers employed transfer learning from a pre-trained model.
Here's a breakdown of the key methodological steps:
- Video Data Collection: Videos of stroke patients undergoing inpatient rehabilitation therapy were recorded using smartphones.
- Region-of-Interest Extraction: The OpenPose library was used for real-time multi-person pose estimation to identify and track individuals. A clinician then manually labeled the stroke patient of interest. This allowed for the extraction of "patient-centered video" and "patient-centered pose" data, effectively isolating the subject and minimizing background noise.
- 3D-CNN for Classification: The patient-centered video segments were fed into the adapted MoViNet-A2 3D-CNN model. This model was modified for a 5 fps input and a 5-second clip length to optimize processing time and quality. The network was trained to classify patients as either dependent or independent in their ambulation.
- Swing Time Asymmetry Analysis: To enhance classification accuracy, particularly in ambiguous cases, the researchers incorporated swing time asymmetry. This metric, derived from the patient's pose keypoints (specifically heel and toe movements), measures the difference in the time it takes for the paretic and non-paretic legs to swing. This feature was used as a supplementary input to the classification model.
Datasets
The study utilized video data collected from 206 stroke patients who received inpatient rehabilitation therapy at Kyungpook National University Chilgok Hospital.
- Data Source: Smartphone videos recorded during physical therapy sessions.
- Total Clips: 2311 video clips were initially collected.
- Walk Action Clips: 1218 clips specifically featuring the "walk" action were selected for analysis, representing approximately 52.7% of the total.
- Patient Demographics: The patients were aged 63.24 ± 14.36 years, with a mix of ischemic and hemorrhagic stroke types.
- Ground Truth: Dependence in ambulation was assessed by physiotherapists using the Functional Ambulatory Categories (FACs) and the Berg Balance Scale (BBS). A FAC score < 4 or a BBS score < 45 was defined as dependent ambulation.
No new benchmark datasets were introduced; the study used existing clinical assessment tools for ground truth.
Key Insights and Ideas
- Smartphone Video as a Viable Data Source: The study demonstrates that readily available smartphone video data, captured in a naturalistic rehabilitation setting, can be effectively used for complex clinical assessments like ambulation dependence classification. This democratizes the technology and makes it more accessible.
- Synergy of 3D-CNN and Kinematic Features: Combining a powerful deep learning model (3D-CNN) with a biomechanical feature (swing time asymmetry) significantly boosted performance. This highlights the benefit of integrating different types of information for more robust predictions.
- Addressing Ambiguity with Kinematic Data: The swing time asymmetry analysis proved particularly useful in cases where the 3D-CNN's output was uncertain (around 0.4-0.6). This suggests that kinematic features can act as a crucial "tie-breaker" for improved diagnostic confidence.
- Privacy-Preserving Potential: By analyzing video data without extracting personal identifying information, the framework offers a privacy-conscious alternative to traditional clinical assessments, which could be critical for adoption in real-world healthcare settings.
Key Results
The study evaluated the performance of the 3D-CNN model using fivefold cross-validation.
- 3D-CNN (FACs and BBS based): Achieved an accuracy of 86.3%, precision of 87.4%, recall of 94.0%, and an F1 score of 90.5%.
- 3D-CNN + Swing Time Asymmetry: This combined approach showed improved performance, reaching 88.7% accuracy, 89.1% precision, 95.7% recall, and a 92.2% F1 score.
- Receiver Operating Characteristic (ROC) Curves: The AUC for dependent ambulation was 0.93 with the 3D-CNN alone and improved to 0.94 when swing time asymmetry was included.
Limitations and Future Work
The authors acknowledge several limitations and suggest avenues for future research:
- Image Noise: The study did not apply image denoising techniques, which could potentially improve performance if the input images are noisy.
- Limited Feature Set: Only swing time asymmetry was analyzed. Future work could explore other spatiotemporal parameters like step length and velocity using 3D pose estimation.
- Video-Only Analysis: The framework currently relies solely on video data. Incorporating audio data from smartphones could provide additional robustness, especially in cases of occlusion.
- Bounding Box vs. Segmentation: While bounding boxes were used for person detection, image segmentation could offer more precise information about patient posture.
Future directions include estimating various spatiotemporal parameters during walking using 3D pose estimation, exploring the integration of audio data, and investigating image segmentation for more detailed posture analysis.
Summary updated