Abstract
In this study, we proposed an on-line reinforcement learning (RL) method suitable for Quasi-Passive Dynamic Walking (Quasi-PDW) by a 2D biped robot, whose possession of knees makes the system unstable. Our study is underlaid by the perspective of low energy consumption and good correspondence to human walking. RL was applied not only for the hip joint but also for the knee joints of the robot, and our learning method was successful in making the unstable robot produce stable walking after as small as 500 training episodes, despite of usage of simple intermittent controllers.
Introduction
Humanoid robotics often faces a trade-off between stability and energy efficiency. Traditional robots, such as ASIMO, rely on the Zero Moment Point (ZMP) criterion, which ensures stability through high-power, continuous motor activity. While effective, this results in "stiff" movements and high energy consumption. Conversely, Passive Dynamic Walking (PDW) utilizes gravity and momentum for extreme efficiency but is highly sensitive to disturbances. Adding knees—essential for human-like gait—further destabilizes PDW systems, often leading to collapse even in static poses.
This research explores Quasi-PDW, a middle ground where small, intermittent torque "nudges" stabilize an otherwise passive gait. The goal is to learn these control parameters using reinforcement learning without requiring an exact mathematical model of the robot's complex dynamics.
Robot Model and Control Framework
The study utilizes a 2D bipedal model within the Open Dynamics Engine (ODE) physics simulator. The robot consists of five links: a torso, two thighs, and two shanks with feet, restricted to the sagittal plane.
To train this unstable system, the authors implemented a two-stage learning approach:
- Initial Condition Search: The robot's knees are initially locked to find a starting posture and velocity that allow for at least seven passive steps. This provides a stable baseline for further learning.
- Reinforcement Learning (RL): Once the knees are unlocked, a stochastic policy gradient method is used to tune an intermittent controller. Instead of continuous feedback, the controller applies rectangular torque pulses to the hip and knees at specific phases of the stride.
The reward function incentivizes a consistent gait (matching states between steps $n$ and $n+1$) and the maintenance of a target step length.
Key Insights
- Intermittent Control: Stability does not require constant intervention. By applying torque only at critical moments—such as the swing-up phase—the RL agent leverages the robot's natural physics.
- Developmental Learning: The transition from locked to unlocked knees mimics biological development, simplifying the high-dimensional learning problem into manageable stages.
- Efficiency by Design: The controller acts as a supplement to passive dynamics, ensuring the resulting gait remains energy-efficient.
Results and Discussion
The proposed RL framework demonstrated rapid convergence, achieving stable walking in approximately 500 episodes. The resulting gait was exceptionally efficient, with a dimension-less cost of transport ($c_{mt}$) of 0.093. This is nearly 17 times more efficient than ZMP-based robots like ASIMO ($c_{mt} \approx 1.6$).
Furthermore, the trained robot exhibited robustness to environmental changes, successfully navigating slopes between 0.02 and 0.04 radians and recovering from impulsive disturbances to the hip joint within a few steps.
Limitations and Future Work
While promising, the current study is limited to 2D sagittal plane simulations. Real-world 3D walking introduces lateral instability, which requires more complex control. Additionally, the current model relies on downslopes for energy; future iterations will focus on level-ground walking where the controller must provide the primary propulsive force. Transitioning these results to physical hardware remains a key objective to address real-world sensor noise and mechanical friction.
Summary updated