Virtual Impedance Adaptation of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning

Ranran Zheng, Zhiyuan Yu, Hongwei Liu, Junqin Lin, Bo Zeng, Longfei JiaChinese 2025 · DOI

TL;DR

This paper introduces VIADRL, a control framework that uses Deep Reinforcement Learning to adaptively tune the "virtual impedance" of a heavy-duty lower-limb exoskeleton. By learning to adjust how the robot responds to human movement in real-time, the system significantly reduces the physical interaction forces between the wearer and the machine, making the exoskeleton feel more transparent and responsive.

What Problem Does This Paper Address?

Lower-limb exoskeletons for human performance augmentation (LEHPA) are designed to help humans carry heavy loads over long distances. However, a major hurdle is the Human-Exoskeleton Interaction (HEI).

Traditional control methods, like Sensitivity Amplification Control (SAC), rely on precise mathematical models of the robot. If the model is slightly off—or as the human’s gait changes with fatigue or speed—the robot can feel "heavy" or resistive. The authors aim to move away from rigid, model-dependent controllers toward an adaptive system that can handle the complex, ever-changing dynamics of a human and robot moving in tandem.

Methods

The authors propose Virtual Impedance Adaptation based on Deep Reinforcement Learning (VIADRL). The core components include:

  • Virtual Impedance Control (VIC): They expanded the classic SAC strategy into a VIC framework. Instead of a single sensitivity factor, they introduced three learnable parameters ($\xi, \zeta, \phi$) that represent virtual mass, damping, and gravity. This allows the controller to fine-tune joint torque with much higher precision.
  • Deep Reinforcement Learning (TD3): The adjustment of these impedance parameters is treated as a Markov Decision Process (MDP). They used the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, an off-policy actor-critic method known for its stability in continuous action spaces.
  • Hybrid Inverse-Forward Dynamics Simulation: To train the agent safely, they built a multibody simulation in MATLAB/Simscape. In this "hybrid" setup, the human model is driven by reference motion data (inverse dynamics), while the exoskeleton joints are modeled in forward dynamics, reacting to the human through simulated spring-damper interaction models.
  • Phase-Dependent Modeling: They partitioned the gait cycle into specific phases (Double Support, Single Support, etc.) based on foot-ground contact to ensure the dynamic model accurately reflected the physics of walking.

Datasets

The study did not use a massive external dataset but rather generated its own training and evaluation data:

  • Reference Motion: Kinematic data (trunk pitch, hip/knee/ankle angles) was captured from a human subject walking on a treadmill at 2.8 km/h.
  • Speed Variation: To broaden the state space, the authors "stretched" and "compressed" this gait cycle to simulate five different walking speeds ranging from 2.8 km/h to 5.6 km/h.
  • Benchmark: They compared their results against SADRL (Sensitivity Adaptation based on DRL), a simpler adaptive strategy from their previous work.

Key Insights and Ideas

  • The "Impedance Cube": The authors visualize the control strategy as a 3D space (Figure 3). While traditional SAC is just a point in this space and SADRL is a line, their VIC strategy allows the agent to explore the entire volume, providing much more flexibility to minimize interaction forces.
  • HEI-Based Reward Function: The DRL agent’s reward is directly tied to minimizing the forces measured at the back, thighs, and shanks. This forces the agent to learn a policy that "gets out of the human's way."
  • Bridging the Reality Gap: The authors acknowledge that simulation isn't perfect. They used a normalization process for state components and HEI forces to successfully transfer the policy trained in simulation to a physical LEHPA prototype.

Key Results

The VIADRL strategy was tested across five walking speeds in both simulation and on a physical prototype:

  • Simulation: The "lumped ratio" of interaction forces was 0.81 compared to the benchmark. This means a 19% reduction in the physical struggle between the human and the robot.
  • Physical Prototype: On the real hardware, the ratio was 0.89, representing an 11% improvement in coordination and a reduction in the load felt by the wearer.
  • Load Transfer: The results showed that VIADRL was particularly effective at transferring the load from the wearer's back and legs directly to the ground, significantly improving the "transparency" of the exoskeleton during the swing phase.

Limitations and Future Work

  • Terrain Constraints: This study focused exclusively on level ground walking. The authors note that future work must involve training the controller on slopes and stairs to handle real-world environments.
  • Dimensionality: The current model focuses on the sagittal plane (forward/backward movement). Lateral (side-to-side) stability and rotation were not addressed in this DRL framework.
  • The Reality Gap: While the transfer to the prototype was successful, it required manual normalization of sensor data. The authors suggest that future iterations could use the DRL framework to fine-tune the controller directly on the physical robot to further close the gap between simulation and reality.

Summary updated

Abstract

Abstract This paper proposes virtual impedance adaptation of the lower-limb exoskeleton for human performance augmentation (LEHPA) based on deep reinforcement learning (VIADRL) to mitigate reliance on model accuracy and address the ever-changing human-exoskeleton interaction (HEI) dynamics. The classical sensitivity amplification control strategy is expanded to the virtual impedance control strategy with more learnable virtual impedance parameters. The adjustment of these virtual impedance parameters is formalized as finding the optimal policy for a Markov Decision Process and can then be effectively resolved using deep reinforcement learning algorithms. To ensure safe and efficient policy training, a multibody simulation environment is established to facilitate the training process, supplemented by the innovative hybrid inverse-forward dynamics simulation approach for executing the simulation. For comparison purposes, the SADRL strategy is introduced as a benchmark. A novel control performance evaluation method based on the HEI forces at the back, thighs, and shanks is proposed to quantitatively evaluate the performance of our proposed VIADRL strategy. The VIADRL controller is systematically compared with the SADRL controller at five selected walking speeds. The lumped ratio of HEI forces under the SADRL strategy relative to those under the SADRL strategy is as low as 0.81 in simulation and approximately 0.89 on the LEHPA prototype. The overall reduction of HEI forces demonstrates the superiority of the VIADRL strategy in comparison to the SADRL strategy.
BibTeX
@inproceedings{zheng2025virtual,
  author = {Ranran Zheng and Zhiyuan Yu and Hongwei Liu and Junqin Lin and Bo Zeng and Longfei Jia},
  title = {{Virtual Impedance Adaptation of Lower-Limb Exoskeleton for Human Performance Augmentation Based on Deep Reinforcement Learning}},
  booktitle = {Chinese},
  year = {2025},
  doi = {10.1186/s10033-025-01355-y},
}
Figure