On-line Human Gait Stability Prediction using LSTMs for the fusion of Deep-based Pose Estimation and LRF-based Augmented Gait State Estimation in an Intelligent Robotic Rollator

Georgia Chalvatzaki, Petros Koutras, Jack Hadfield, Xanthi S. Papageorgiou, Costas S. Tzafestas, Petros Maragos

Abstract

In this work, we present a novel framework for on-line human gait stability prediction of the elderly users of an intelligent robotic rollator using Long Short Term Memory (LSTM) networks, fusing multimodal RGB-D and Laser Range Finder (LRF) data from non-wearable sensors. A Deep Learning (DL) based approach is used for the upper body pose estimation. The detected pose is used for estimating the body Center of Mass (CoM) using Unscented Kalman Filter (UKF). An Augmented Gait State Estimation framework exploits the LRF data to estimate the legs' positions and the respective gait phase. These estimates are the inputs of an encoder-decoder sequence to sequence model which predicts the gait stability state as Safe or Fall Risk walking. It is validated with data from real patients, by exploring different network architectures, hyperparameter settings and by comparing the proposed method with other baselines. The presented LSTM-based human gait stability predictor is shown to provide robust predictions of the human stability state, and thus has the potential to be integrated into a general user-adaptive control architecture as a fall-risk alarm.

Introduction

Falls represent a significant health risk for the elderly, often leading to severe injury and a loss of independence. Approximately 30% of individuals over the age of 65 experience at least one fall annually. While robotic rollators (smart walkers) offer physical support, most current systems are passive observers of stability.

Predicting falls is challenging due to the complex, non-linear relationship between a person's Center of Mass (CoM) and their Base of Support (BoS)—the area between their feet. Traditional methods rely on wearable sensors or motion-capture laboratories, which are impractical for daily use. This research introduces a framework that utilizes only "non-wearable" sensors mounted directly on a robotic rollator to predict gait stability in real-time, allowing the robot to potentially intervene before a fall occurs.

The robotic rollator system classifying gait as Safe or Fall Risk in a clinical setting.

System Architecture and Methodology

The proposed framework employs a multi-stage pipeline that integrates classical robotics estimation with deep learning to process multimodal sensor data.

Multimodal Data Fusion

The system fuses data from two primary sources:

  1. Upper Body Pose & CoM Estimation: An RGB-D camera (Kinect) captures the user's upper body. The OpenPose library detects 2D keypoints, which are then projected into 3D space. To handle frequent occlusions caused by the camera's proximity to the user, an Unscented Kalman Filter (UKF) provides a smoothed, continuous estimate of the CoM.
  2. Leg Tracking & Gait Phase: A Laser Range Finder (LRF) scans the user’s legs at knee height. An "Augmented Gait State Estimation" framework, utilizing Particle Filters, tracks leg positions and identifies gait phases (e.g., "Left Swing" or "Double Support").

LSTM-based Prediction Network

The core of the stability prediction is an encoder-decoder sequence-to-sequence model. The architecture includes:

  • Feature Encoding: Two Fully Connected (FC) layers transform raw CoM and leg data into high-level representations. This step is crucial for capturing the complex interactions between the torso and legs.
  • Temporal Modeling: Two Long Short-Term Memory (LSTM) layers process a sliding window of 100 frames (approximately 3 seconds) to capture movement dynamics.
  • Classification: A final Softmax layer classifies the gait state as "Safe" or "Fall Risk."
Overview of the proposed framework, showing the fusion of RGB-D and LRF data into the LSTM-based prediction network.

Experimental Evaluation

The system was validated using clinical data from five patients with mild to moderate mobility impairments at the Agaplesion Bethanien Hospital Geriatric Center. Ground truth for stability was established using a high-precision Motion Capture (MoCap) system to calculate the true stability margin.

Results

The best-performing architecture (FC1+FC2+2-layer LSTM) achieved a mean Area Under the Curve (AUC) of 90.01% and an F-Score of 86.79%. Key findings include:

  • Encoding Necessity: Models without the initial FC layers performed significantly worse (AUC ~67%), highlighting the importance of non-linear feature encoding before temporal processing.
  • Temporal Advantage: The LSTM-based approach outperformed traditional baselines like Support Vector Machines (SVMs), which achieved an AUC of 86.73%.
  • Robustness: The integration of UKF allowed the system to remain robust against sensor occlusions, a common challenge in mobile robotics.

Conclusion and Future Work

This work demonstrates a robust, non-wearable solution for real-time gait stability prediction. By grounding the model in biomechanical principles—specifically the CoM-BoS relationship—the system provides interpretable and accurate fall-risk assessments.

Future research will focus on expanding the dataset to include more diverse pathological gaits and optimizing the model for low-power mobile hardware. Ultimately, these predictions will be integrated into active control strategies, enabling the rollator to intervene (e.g., by braking) to prevent falls before they occur.

Summary updated

Abstract

In this work, we present a novel framework for on-line human gait stability prediction of the elderly users of an intelligent robotic rollator using Long Short Term Memory (LSTM) networks, fusing multimodal RGB-D and Laser Range Finder (LRF) data from non-wearable sensors. A Deep Learning (DL) based approach is used for the upper body pose estimation. The detected pose is used for estimating the body Center of Mass (CoM) using Unscented Kalman Filter (UKF). An Augmented Gait State Estimation framework exploits the LRF data to estimate the legs' positions and the respective gait phase. These estimates are the inputs of an encoder-decoder sequence to sequence model which predicts the gait stability state as Safe or Fall Risk walking. It is validated with data from real patients, by exploring different network architectures, hyperparameter settings and by comparing the proposed method with other baselines. The presented LSTM-based human gait stability predictor is shown to provide robust predictions of the human stability state, and thus has the potential to be integrated into a general user-adaptive control architecture as a fall-risk alarm.
BibTeX
@misc{chalvatzaki2018line,
  author = {Georgia Chalvatzaki and Petros Koutras and Jack Hadfield and Xanthi S. Papageorgiou and Costas S. Tzafestas and Petros Maragos},
  title = {{On-line Human Gait Stability Prediction using LSTMs for the fusion of Deep-based Pose Estimation and LRF-based Augmented Gait State Estimation in an Intelligent Robotic Rollator}},
  year = {2018},
  eprint = {1812.00252},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.00252},
}
Figure