Combining Real-Time Segmentation and Classification of Rehabilitation Exercises with LSTM Networks and Pointwise Boosting

Antonio Bevilacqua, Giovanni Ciampi, Rob Argent, Brian Caulfield, M. Tahar KechadiAAAI 2020 · DOI

Introduction

In physical rehabilitation, patient adherence and the quality of exercise execution are paramount for successful recovery. While autonomous biofeedback systems offer a promising solution for home-based therapy, existing tools often suffer from high latency and fragility. Most current systems employ a disjointed two-step pipeline: first segmenting a continuous sensor stream into discrete repetitions, and then classifying those repetitions. This approach often fails in clinical settings where patient movements are noisy, leading to "oversegmentation"—where minor tremors or pauses are incorrectly identified as the end of a movement.

This paper presents a novel deep learning framework designed for real-time segmentation and qualitative classification of rehabilitation exercises. By integrating these two steps into a cohesive process using Long Short-Term Memory (LSTM) networks, the system provides immediate, robust feedback to patients without requiring pre-defined templates or domain-specific feature engineering.

Proposed Framework

The core of the proposed system is a pipeline that processes multivariate time-series data from wearable inertial sensors (acceleration and angular velocity). Unlike traditional window-based classifiers, this architecture produces pointwise predictions, allowing for finer granularity and lower latency.

The proposed real-time processing pipeline, featuring a pointwise LSTM-CNN classifier followed by SMA-based segmentation and boosting aggregation.

Pointwise Prediction and Segmentation

The system utilizes an LSTM network to generate a probability vector for every individual timestep. This vector classifies the current state into one of three categories: silent (rest), correct movement, or deviant (incorrect) movement.

To address the inherent jitter in raw neural network outputs, the researchers apply a Simple Moving Average (SMA) to the "silent" probability stream. This acts as a low-pass filter; when the smoothed probability drops below a specific threshold, the system triggers the start of a movement segment. Conversely, when it rises back above the threshold, the segment is closed. This method effectively prevents oversegmentation during the hesitant movements typical of clinical patients.

Prediction Boosting

Once a segment is isolated, the system must aggregate the pointwise probabilities into a single qualitative label for the entire repetition. The authors evaluated several "boosting" strategies, including Deviation Voting (majority vote), Cumulative Deviation Strength (summing probabilities), and q-best Deviation Rank (qDR). The qDR method, which considers the most frequent class among the top q highest probability timesteps, was found to be the most effective for final classification.

Experimental Evaluation

The framework was tested on data from four common lower-limb exercises: Heel Slide (HS), Seated Knee Extension (SKE), Inner Range Quadriceps (IRQ), and Straight Leg Raise (SLR). The study included both healthy subjects and a clinical test set of actual patients, providing realistic examples of aberrant movements like tremors and co-articulations.

Average classification accuracy across different exercises using the Leave-One-Subject-Out (LOSO) cross-validation method.

Key Results

  • Segmentation Robustness: The LSTM-based segmentation significantly outperformed traditional methods. On the clinical dataset for Heel Slides, the system achieved 84.6% accuracy, whereas traditional Zero Velocity Crossing (ZVC) achieved only 55%.
  • Classification Accuracy: The system demonstrated high precision in distinguishing correct from deviant movements. For healthy subjects, classification accuracy reached 100% for Seated Knee Extension and over 96% for Heel Slides.
  • Clinical Viability: The model proved robust against the noisy movement patterns of clinical patients, where "correct" movements often appear more erratic than the "incorrect" movements of healthy subjects.

Conclusion and Future Work

By intertwining segmentation and classification through pointwise LSTM predictions, this framework overcomes the latency and fragility issues of previous state-of-the-art systems. While currently limited to binary (correct/deviant) classification, future work aims to implement hierarchical models to provide specific corrective feedback (e.g., identifying specific postural errors) and to expand the system's application to complex upper-limb movements.

Summary updated

BibTeX
@inproceedings{bevilacqua2020combining,
  author = {Antonio Bevilacqua and Giovanni Ciampi and Rob Argent and Brian Caulfield and M. Tahar Kechadi},
  title = {{Combining Real-Time Segmentation and Classification of Rehabilitation Exercises with LSTM Networks and Pointwise Boosting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i08.7028},
}
Figure