Abstract
In this work, we present a novel framework for on-line human gait stability prediction of the elderly users of an intelligent robotic rollator using Long Short Term Memory (LSTM) networks, fusing multimodal RGB-D and Laser Range Finder (LRF) data from non-wearable sensors. A Deep Learning (DL) based approach is used for the upper body pose estimation. The detected pose is used for estimating the body Center of Mass (CoM) using Unscented Kalman Filter (UKF). An Augmented Gait State Estimation framework exploits the LRF data to estimate the legs' positions and the respective gait phase. These estimates are the inputs of an encoder-decoder sequence to sequence model which predicts the gait stability state as Safe or Fall Risk walking. It is validated with data from real patients, by exploring different network architectures, hyperparameter settings and by comparing the proposed method with other baselines. The presented LSTM-based human gait stability predictor is shown to provide robust predictions of the human stability state, and thus has the potential to be integrated into a general user-adaptive control architecture as a fall-risk alarm.
Introduction
Falls are a leading cause of injury and loss of independence among the elderly. While robotic rollators (smart walkers) provide physical support, most current versions are passive observers regarding stability. Predicting gait stability in real-time is a significant challenge due to sensor limitations (wearables are cumbersome), frequent occlusions in close-range camera setups, and the highly non-linear relationship between upper body sway and leg positioning during different gait phases.
Methodology
The researchers proposed a multi-stage pipeline that fuses data from two non-wearable sensors to track the interaction between the body's Center of Mass (CoM) and its Base of Support (BoS):
- Upper Body Tracking (RGB-D): A Kinect camera captures the user's upper body. The system uses OpenPose for 2D keypoint detection, maps them to 3D using depth data, and employs an Unscented Kalman Filter (UKF) to smooth noisy data and handle occlusions.
- Lower Body Tracking (LRF): A Laser Range Finder (LRF) scans the user's legs at knee height. An Augmented Gait State Estimation framework using particle filters tracks leg positions and identifies the current gait phase.
- Stability Prediction (LSTM): The estimated CoM and leg positions are fed into a sequence-to-sequence model. The architecture uses two Fully Connected (FC) layers as an encoder to find a robust feature representation, followed by two Long Short-Term Memory (LSTM) layers to capture temporal dependencies. The final output is a binary classification: Safe or Fall Risk.
Experimental Evaluation and Results
The study utilized a custom dataset of approximately 11,000 frames collected from five elderly patients with mobility impairments at the Agaplesion Bethanien Hospital Geriatric Center. Ground truth was established using a high-precision Motion Capture (MoCap) system.
Key findings include:
- Architecture Performance: The
FC1+FC2+LSTMmodel achieved an Area Under the Curve (AUC) of 90.01% and an F-Score of 86.79%, significantly outperforming plain LSTM baselines (AUC ~59%). - The Importance of Encoding: Adding fully connected layers before the LSTM was critical, suggesting the network needs to learn a static mapping of the CoM-BoS relationship before modeling gait timing.
- Temporal Windowing: A sequence of 100 instances (roughly 3 seconds of data) provided the most consistent performance across all metrics.
Discussion and Future Work
While the results are promising, the study noted performance drops for specific users with highly pathological gaits, suggesting a need for user-specific calibration. The next phase of research involves "closing the loop" by integrating this prediction framework into an active control architecture. Once a fall risk is predicted, the rollator could proactively intervene—for example, by applying brakes or providing haptic feedback—to prevent an accident before it occurs.
Summary updated