Continuous Prediction of Lower-Limb Kinematics From Multi-Modal Biomedical Signals

Chunzhi Yi, Feng Jiang, Shengping Zhang, Hao Guo, Chifu Yang, Zhen Ding, Baichun Wei, Xiangyuan Lan, Huiyu Zhou

TL;DR

Researchers have developed KinPreNet, a deep learning model that predicts lower-limb joint angles up to 100ms into the future by fusing muscle activity (EMG) and motion sensor (IMU) data. By leveraging the natural physiological delay between muscle activation and movement, this system can compensate for the mechanical lag in exoskeletons, leading to smoother, more intuitive robotic assistance.

What Problem Does This Paper Address?

Lower-limb exoskeletons are promising tools for rehabilitation and mobility, but they suffer from a frustrating "lag." This lag is caused by the mechanical transmission of the robot—the time it takes for motors and gears to actually move once a command is sent. If the robot is even a few dozen milliseconds behind the wearer, the assistance feels clunky, heavy, or even unsafe.

To solve this, the robot needs to be a "mind reader." It must predict what the wearer is about to do before they do it. While previous research could identify discrete gait phases (like "heel strike") or predict just one step ahead, these methods weren't continuous or long-range enough to fully mask the mechanical delay. The challenge lies in creating a continuous, ahead-of-time prediction of joint kinematics that is robust enough for real-world use.

Methods

The authors propose an end-to-end architecture called KinPreNet, which consists of two main parts:

  1. LSTM Feature Extractor: Instead of using hand-crafted mathematical features from Electromyography (EMG) signals, they used a Long Short-Term Memory (LSTM) network to automatically learn the complex, non-linear relationships between the firing of nine different muscles.
  2. LSTM Motion Predictor: This takes the extracted muscle features and current knee angles (from IMUs) to predict the knee angle at a specific future time point.

The researchers used a "Teacher-Student" paradigm for training. A high-precision stereo-photogrammetric system (Vicon) acted as the "Teacher," providing the "gold standard" ground truth for joint angles. The KinPreNet (the "Student") learned to replicate these angles using only wearable sensors (EMG and IMUs).

Datasets

The study involved:

  • Subjects: 10 healthy individuals (8 men, 2 women).
  • Sensors: Surface EMG electrodes on 9 muscles (including the rectus femoris, vastus lateralis, and gastrocnemius) and 9-axis IMUs.
  • Ground Truth: A Vicon 8-camera motion capture system tracking 16 retro-reflective markers.
  • Data Volume: Over 90 million EMG sample points and 670,000 IMU-calculated knee angle points collected during self-selected walking speeds.

Key Insights and Ideas

  • The EMD Advantage: The "aha moment" of this paper is the exploitation of Electromechanical Delay (EMD). In humans, there is a natural 25–125ms gap between when a muscle electrically activates and when the limb actually moves. By tapping into this physiological buffer, the researchers found they could predict motion far enough in advance to cancel out robotic mechanical lag.
  • End-to-End beats Hand-Crafted: The study compared their learned LSTM features against traditional time-domain features (like Mean Absolute Value or Zero Crossings). The learned features significantly outperformed the traditional ones, suggesting that deep learning can capture muscle synergy and antagonism that simple math misses.
  • The "Sweet Spot" for Prediction: The researchers tested six different prediction windows. They discovered a performance "boundary": predictions were highly accurate between 27ms and 108ms. Beyond 135ms, the accuracy dropped significantly, defining the physical limits of how far ahead we can reliably "see" using muscle signals.

Key Results

  • High Accuracy: KinPreNet achieved an average Root Mean Square Error (RMSE) of 3.98 degrees. For context, traditional ahead-of-time prediction methods resulted in a much higher error of 15.95 degrees.
  • Superior Correlation: The model maintained a high Pearson correlation coefficient (R-value of 0.88) even when predicting 100ms into the future.
  • Real-time Potential: The inference time for the algorithm is only 5–7ms, making it fast enough to run on embedded hardware like the NVIDIA TX2 commonly used in mobile robotics.

Limitations and Future Work

While the results are impressive, the authors acknowledge a few hurdles:

  • Healthy Subjects Only: The model was trained on healthy individuals. People requiring exoskeletons (e.g., stroke survivors or those with paraplegia) often have altered EMG patterns or muscle atrophy, which may require personalized model tuning.
  • Controlled Environment: The study focused on level walking at self-selected speeds. Future work needs to address more complex scenarios like stair climbing, ramps, or sudden changes in pace.
  • Generalization: The current study focuses on the knee joint; extending this to the hip and ankle simultaneously will be necessary for a full-body assistive suit.

Summary updated

Abstract

The fast-growing techniques of measuring and fusing multi-modal biomedical signals enable advanced motor intent decoding schemes of lower-limb exoskeletons, meeting the increasing demand for rehabilitative or assistive applications of take-home healthcare. Challenges of exoskeletons’ motor intent decoding schemes remain in making a continuous prediction to compensate for the hysteretic response caused by mechanical transmission. In this paper, we solve this problem by proposing an ahead-of-time continuous prediction of lower-limb kinematics, with the prediction of knee angles during level walking as a case study. Firstly, an end-to-end kinematics prediction network(KinPreNet), <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><sup>1</sup></xref> consisting of a feature extractor and an angle predictor, is proposed and experimentally compared with features and methods traditionally used in ahead-of-time prediction of gait phases. Secondly, inspired by the electromechanical delay(EMD), we further explore our algorithm’s capability of compensating response delay of mechanical transmission by validating the performance of the different sections of prediction time. And we experimentally reveal the time boundary of compensating the hysteretic response. Thirdly, a comparison of employing EMG signals or not is performed to reveal the EMG and kinematic signals’ collaborated contributions to the continuous prediction. During the experiments, EMG signals of nine muscles and knee angles calculated from inertial measurement unit (IMU) signals are recorded from ten healthy subjects. Our algorithm can predict knee angles with the averaged RMSE of 3.98 deg which is better than the 15.95-deg averaged RMSE of utilizing the traditional methods of ahead-of-time prediction. The best prediction time is in the interval of 27ms and 108ms. To the best of our knowledge, this is the first study of continuously predicting lower-limb kinematics in an ahead-of-time manner based on the electromechanical delay (EMD).
BibTeX
@article{yi2022continuous,
  author = {Chunzhi Yi and Feng Jiang and Shengping Zhang and Hao Guo and Chifu Yang and Zhen Ding and Baichun Wei and Xiangyuan Lan and Huiyu Zhou},
  title = {{Continuous Prediction of Lower-Limb Kinematics From Multi-Modal Biomedical Signals}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2022},
  url = {https://openreview.net/forum?id=bcVThZZ-uKM},
}

Loading related content...

Figure