Reinforcement Learning of Stable Trajectory for Quasi-Passive Dynamic Walking of an Unstable Biped Robot

Tomohiro Shibata, Kentarou Hitomoi, Yutaka Nakamura, Shin IshiiOpenalex Unknown 2007 · DOI · OpenAlex

Abstract

In this study, we proposed an on-line reinforcement learning (RL) method suitable for Quasi-Passive Dynamic Walking (Quasi-PDW) by a 2D biped robot, whose possession of knees makes the system unstable. Our study is underlaid by the perspective of low energy consumption and good correspondence to human walking. RL was applied not only for the hip joint but also for the knee joints of the robot, and our learning method was successful in making the unstable robot produce stable walking after as small as 500 training episodes, despite of usage of simple intermittent controllers.

Introduction

Humanoid robotics often faces a trade-off between stability and energy efficiency. Traditional robots, such as ASIMO, rely on the Zero Moment Point (ZMP) criterion, which ensures stability through high-power, continuous motor activity. While effective, this results in "stiff" movements and high energy consumption. Conversely, Passive Dynamic Walking (PDW) utilizes gravity and momentum for extreme efficiency but is highly sensitive to disturbances. Adding knees—essential for human-like gait—further destabilizes PDW systems, often leading to collapse even in static poses.

This research explores Quasi-PDW, a middle ground where small, intermittent torque "nudges" stabilize an otherwise passive gait. The goal is to learn these control parameters using reinforcement learning without requiring an exact mathematical model of the robot's complex dynamics.

Robot Model and Control Framework

The study utilizes a 2D bipedal model within the Open Dynamics Engine (ODE) physics simulator. The robot consists of five links: a torso, two thighs, and two shanks with feet, restricted to the sagittal plane.

The 2D biped robot model with hip and knee joints.

To train this unstable system, the authors implemented a two-stage learning approach:

  1. Initial Condition Search: The robot's knees are initially locked to find a starting posture and velocity that allow for at least seven passive steps. This provides a stable baseline for further learning.
  2. Reinforcement Learning (RL): Once the knees are unlocked, a stochastic policy gradient method is used to tune an intermittent controller. Instead of continuous feedback, the controller applies rectangular torque pulses to the hip and knees at specific phases of the stride.
Intermittent torque control pulses applied during the gait cycle.

The reward function incentivizes a consistent gait (matching states between steps $n$ and $n+1$) and the maintenance of a target step length.

Key Insights

  • Intermittent Control: Stability does not require constant intervention. By applying torque only at critical moments—such as the swing-up phase—the RL agent leverages the robot's natural physics.
  • Developmental Learning: The transition from locked to unlocked knees mimics biological development, simplifying the high-dimensional learning problem into manageable stages.
  • Efficiency by Design: The controller acts as a supplement to passive dynamics, ensuring the resulting gait remains energy-efficient.

Results and Discussion

The proposed RL framework demonstrated rapid convergence, achieving stable walking in approximately 500 episodes. The resulting gait was exceptionally efficient, with a dimension-less cost of transport ($c_{mt}$) of 0.093. This is nearly 17 times more efficient than ZMP-based robots like ASIMO ($c_{mt} \approx 1.6$).

Furthermore, the trained robot exhibited robustness to environmental changes, successfully navigating slopes between 0.02 and 0.04 radians and recovering from impulsive disturbances to the hip joint within a few steps.

Limitations and Future Work

While promising, the current study is limited to 2D sagittal plane simulations. Real-world 3D walking introduces lateral instability, which requires more complex control. Additionally, the current model relies on downslopes for energy; future iterations will focus on level-ground walking where the controller must provide the primary propulsive force. Transitioning these results to physical hardware remains a key objective to address real-world sensor noise and mechanical friction.

Summary updated

Abstract

In this study, we proposed an on-line RL method suitable for Quasi-PDW by a 2D biped robot, whose possession of knees makes the system unstable. Our study is underlaid by the perspective of low energy consumption and good correspondence to human walking. RL was applied not only for the hip joint but also for the knee joints of the robot, and our learning method was successful in making the unstable robot produce stable walking after as small as 500 training episodes, despite of usage of simple intermittent controllers. Although the controller itself was simple, simulation experiments on downslopes with
BibTeX
@inproceedings{shibata2007reinforcement,
  author = {Tomohiro Shibata and Kentarou Hitomoi and Yutaka Nakamura and Shin Ishii},
  title = {{Reinforcement Learning of Stable Trajectory for Quasi-Passive Dynamic Walking of an Unstable Biped Robot}},
  booktitle = {Openalex Unknown},
  year = {2007},
  doi = {10.5772/4804},
}

Loading related content...

Figure