Depth-aware pose estimation using deep learning for exoskeleton gait analysis

Yachun Wang, Zhongcai Pei, Chen Wang, Zhiyong TangScientific 2023 · DOI

TL;DR

Researchers have developed GaitPoseNet, a non-contact, vision-based system that uses deep learning to monitor the gait of patients wearing lower-limb rehabilitation exoskeletons. By combining implicit depth estimation with a novel "Joint Guidance Strategy," the system achieves high accuracy (95.77% PCK) and blazing-fast performance (3.55 ms per frame), making real-time clinical gait analysis possible without uncomfortable wearable sensors.

What Problem Does This Paper Address?

In rehabilitation medicine, monitoring a patient’s gait while they use a lower-limb exoskeleton is critical to prevent falls and ensure symmetrical movement. Traditionally, this is done using wearable sensors like IMUs (Inertial Measurement Units) or EMG (electromyography) pads.

However, these "contact-based" methods have significant drawbacks: they can be uncomfortable, they often interfere with a patient's natural walking motion, and they require professional technicians to set up and interpret the data. Computer vision offers a "non-contact" alternative, but it faces a major hurdle: occlusion. The bulky metal frames of the exoskeleton often block the camera's view of the human joints, leading to "lost" data points in standard pose estimation models.

Methods

The authors propose a multi-task convolutional neural network (CNN) architecture called GaitPoseNet. Its key innovations include:

  • Implicit Depth Supervision: Unlike many models that require expensive RGB-D (depth) cameras as input, GaitPoseNet takes a standard RGB image but is trained to predict a depth map as an auxiliary output. This "depth-awareness" helps the model better understand the 3D geometric relationship between joints, even though it only outputs 2D coordinates during inference.
  • Joint Guidance Strategy (JGS): To solve the occlusion problem, the network predicts the directional features (the "bones") between pairs of joints. If a knee is hidden by a metal strut, the model uses the position of the hip and the predicted direction of the thigh bone to "guide" itself to the most likely location of the occluded knee.
  • Geometric Post-Processing: The system converts 2D pixel coordinates into 3D gait parameters (like hip/knee rotation angles and stride length) by combining the visual data with the patient’s known leg length and a pinhole camera model.

Datasets

The study utilized three main datasets:

  1. COCO & LSP: Standard large-scale human pose datasets used for initial pre-training.
  2. Walking Pose with Exoskeleton (WPE) Dataset: A new benchmark introduced by the authors. It contains 7,176 RGB images and corresponding depth maps of 10 subjects walking freely while wearing an exoskeleton in various environments. This is a significant contribution, as most public datasets lack the specific "human-machine" visual complexity of exoskeleton use.

Key Insights and Ideas

  • Depth as an Output, Not an Input: One of the most clever moves here is treating depth as a training target rather than a required input. This allows the final deployed system to run on cheap, standard cameras while still benefiting from the spatial "intelligence" that depth data provides.
  • Connectivity Matters: By focusing on the correlation between joints (JGS) rather than just treating joints as isolated points, the model mimics how a human observer might guess where a hidden limb is based on the rest of the body's posture.
  • Efficiency for Mobile Health: The authors developed a "Lite" version of the model. By adjusting the network channel width, they achieved a massive speedup with only a negligible drop in accuracy, proving that high-end gait analysis can run on mobile laptops in a clinical setting.

Key Results

  • Accuracy: The model achieved a 95.77% [email protected] (Percentage of Correct Keypoints) on the WPE dataset.
  • Speed: The "Lite" version boasts a runtime of just 3.55 ms per frame on a mobile GPU. This is roughly 280 frames per second, far exceeding the requirements for real-time monitoring.
  • Validation: When compared against the HTC Vive tracker (a high-precision, laser-based motion capture system), GaitPoseNet’s joint angle curves showed extremely high similarity, validating its use as a clinical-grade tool.

Limitations and Future Work

While the system is highly effective in the sagittal (side) plane, the authors acknowledge a few limitations:

  • Relative vs. Absolute Depth: The current model predicts relative depth features. Because it doesn't calculate absolute distance in world coordinates, it cannot yet perfectly map the subject's behavior in 3D space for complex maneuvers like turning.
  • Viewpoint Dependency: The accuracy is highest when the camera is unobstructed. While JGS helps with partial occlusion, total occlusion of a limb remains a challenge.
  • Future Directions: The team plans to evolve the network to regress coordinates directly into a 3D world coordinate system, enabling analysis of more complex walking patterns beyond straight-line rehabilitation.

Summary updated

Abstract

In rehabilitation medicine, real-time analysis of the gait for human wearing lower-limb exoskeleton rehabilitation robot during walking can effectively prevent patients from experiencing excessive and asymmetric gait during rehabilitation training, thereby avoiding falls or even secondary injuries. To address the above situation, we propose a gait detection method based on computer vision for the real-time monitoring of gait during human-machine integrated walking. Specifically, we design a neural network model called GaitPoseNet, which is used for posture recognition in human-machine integrated walking. Using RGB images as input and depth features as output, regression of joint coordinates through depth estimation of implicit supervised networks. In addition, joint guidance strategy (JGS) is designed in the network framework. The degree of correlation between the various joints of the human body is used as a detection target to effectively overcome prediction difficulties due to partial joint occlusion during walking. Finally, a post processing algorithm is designed to describe patients' walking motion by combining the pixel coordinates of each joint point and leg length. Our advantage is that we provide a non-contact measurement method with strong universality, and use depth estimation and JGS to improve measurement accuracy. Conducting experiments on the Walking Pose with Exoskeleton (WPE) Dataset shows that our method can reach 95.77% [email protected], 93.14% [email protected] and 3.55 ms runtime. Therefore our method achieves advanced performance considering both speed and accuracy.
BibTeX
@inproceedings{wang2023depth,
  author = {Yachun Wang and Zhongcai Pei and Chen Wang and Zhiyong Tang},
  title = {{Depth-aware pose estimation using deep learning for exoskeleton gait analysis}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-50207-z},
}
Figure