End-to-End High-Level Control of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning

Ranran Zheng, Zhiyuan Yu, Hongwei Liu, Jing Chen, Zhe Zhao, Longfei JiaIEEE Access 2023 · DOI

This paper, published in IEEE Access, presents a significant step forward in making wearable robots feel less like heavy machinery and more like a natural extension of the human body. By leveraging Deep Reinforcement Learning (DRL), the researchers from the Beijing Institute of Technology have developed a control system that bypasses the "math-heavy" traditional approach to exoskeleton movement.

TL;DR

Researchers developed a model-free, end-to-end controller for lower-limb exoskeletons that uses Deep Reinforcement Learning to predict human motion intention. By training in a specialized simulation environment to minimize interaction forces, the system achieved a 35% reduction in resistance compared to unpowered modes, significantly improving wearer "transparency."

What Problem Does This Paper Address?

The holy grail of exoskeleton design is transparency: the ability of the robot to move so perfectly in sync with the wearer that they feel zero resistance. Current Lower-limb Exoskeletons for Human Performance Augmentation (LEHPA) struggle with this.

Traditional control strategies usually rely on complex kinematic and dynamic models that are difficult to calibrate and often fail to account for the messy, non-linear nature of human-robot interaction. Other methods use physiological signals like sEMG (muscle sensors), but these are notoriously noisy and require complex setups. The authors address this gap by proposing a "model-free" approach that learns to "feel" the wearer’s intent through force sensors rather than calculating it through rigid geometry.

Methods

The authors propose a two-level control architecture called E2EDRL (End-to-End Deep Reinforcement Learning):

  1. High-Level Control (The "Brain"): Uses a Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. This neural network takes in 25 dimensions of data—including exoskeleton joint angles, velocities, and 11 different human-exoskeleton interaction (HEI) force components—to predict the wearer's intended movement.
  2. Low-Level Control (The "Muscles"): Standard PD (Proportional-Derivative) controllers that take the high-level "intent" and translate it into actual motor torques.

The Simulation Innovation: Training a robot while a human is wearing it is dangerous and slow. To solve this, the authors built a novel multibody simulation environment in MATLAB/Simscape. They used a hybrid inverse-forward dynamics method: the "human" model is driven by pre-recorded motion data (inverse dynamics), while the "exoskeleton" reacts to the human via simulated force sensors (forward dynamics). This allowed the agent to learn safely and efficiently in a virtual space.

Datasets

  • Human Motion Data: The researchers used gait cycle data (trunk, hip, knee, and ankle trajectories) collected from a human walking at 2.8 km/h using an Xsens motion capture system.
  • Synthetic Benchmarks: To broaden the agent's experience, they stretched and compressed this gait data to simulate walking speeds ranging from 2.8 km/h to 5.6 km/h.
  • Baseline: A "Passive Mode" (where the exoskeleton is worn but unpowered) was used as the primary benchmark to measure how much the active controller reduced resistance.

Key Insights and Ideas

  • HEI as the Reward Signal: Instead of just trying to match a trajectory, the DRL agent is rewarded for minimizing interaction forces. If the robot pushes against the human, it gets a penalty. This directly optimizes for wearer comfort.
  • Action Augmentation: Rather than outputting raw joint angles, the policy learns to output augmentations ($\Delta\theta$) to the current state. This "delta-based" approach makes the learning process much more stable and efficient.
  • Bypassing Kinematics: By using an end-to-end neural network, the system doesn't need to know the exact length of the wearer's limbs or the mass of the robot segments to function effectively. It learns the relationship between force and motion implicitly.

Key Results

The researchers evaluated the system using a "Global Ratio" ($\lambda^*$) of interaction forces.

  • 35% Reduction in Resistance: The E2EDRL strategy achieved a global force ratio of 0.65 compared to the passive mode. This means the wearer feels 35% less "drag" from the robot.
  • Superior Payload Transfer: Significant reductions in pitch torque and vertical force at the back interface indicate that the exoskeleton successfully transferred the payload weight to the ground without the wearer having to fight the machine.
  • Speed Adaptability: The controller remained effective across a wide range of walking speeds (2.8–5.6 km/h), though the authors noted that the shank interface is the most difficult to optimize due to the complex transition between stance and swing phases.

Limitations and Future Work

While the results are promising, the study was conducted entirely in a simulated environment. The authors acknowledge the "reality gap"—the difference between a simulation and the real world—as a primary hurdle.

Future directions include: * Real-world Implementation: Testing the E2EDRL controller on a physical LEHPA platform to see how it handles real-world sensor noise and mechanical friction. * Complex Terrains: Moving beyond flat ground to train the agent on stairs, slopes, and uneven surfaces. * User Adaptation: Training the system to adapt to different human gait styles and body proportions in real-time.

Summary updated

Abstract

This paper proposes a novel end-to-end controller for the lower-limb exoskeleton for human performance augmentation (LEHPA) systems based on deep reinforcement learning (E2EDRL). The model-free controller contains two control levels: the high-level control responsible for end-to-end human motion intention recognition based on the exoskeleton state signals and the human-exoskeleton interaction (HEI) force signals by deep neural network predictor, and the low-level control for motion tracking by joint PD controllers. The deep neural network predictor does not require complex kinematic calculations that are inevitable in conventional human motion intention recognition methods. We execute the learning process in simulation to learn the E2EDRL strategy efficiently and safely by constructing a novel multibody simulation environment and proposing its specific hybrid inverse-forward dynamics simulation method. The passive mode (all joints remain unpowered) is introduced as a benchmark for comparison purposes. A novel performance assessment method based on HEI forces is put forward to evaluate the E2EDRL strategy quantitatively. The global ratio of the HEI forces in the E2EDRL strategy relative to those in the passive mode is as low as 0.65. The global reduction of the HEI forces demonstrates the superior control performance of the E2EDRL strategy.
BibTeX
@inproceedings{zheng2023end,
  author = {Ranran Zheng and Zhiyuan Yu and Hongwei Liu and Jing Chen and Zhe Zhao and Longfei Jia},
  title = {{End-to-End High-Level Control of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning}},
  booktitle = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3317183},
}
Figure