Abstract
Bipedal walking is one of the most difficult but exciting challenges in robotics. The difficulties arise from the complexity of high-dimensional dynamics, sensing and actuation limitations combined with real-time and computational constraints. Deep Reinforcement Learning (DRL) holds the promise to address these issues by fully exploiting the robot dynamics with minimal craftsmanship. In this paper, we propose a novel DRL approach that enables an agent to learn omnidirectional locomotion for humanoid (bipedal) robots. Notably, the locomotion behaviors are accomplished by a single control policy (a single neural network). We achieve this by introducing a new curriculum learning method that gradually increases the task difficulty by scheduling target velocities. In addition, our method does not require reference motions which facilities its application to robots with different kinematics, and reduces the overall complexity. Finally, different strategies for sim-to-real transfer are presented which allow us to transfer the learned policy to a real humanoid robot.
Introduction
Bipedal walking is a "grand challenge" in robotics due to high-dimensional dynamics, unstable equilibrium, and strict real-time constraints. Traditionally, engineers used complex mathematical models or "stitched together" multiple controllers for different speeds and directions. While DRL has shown promise, most existing methods rely on reference motions (mimicking human data) or engineered modules (like trajectory planners). These dependencies make it hard to adapt the gait to robots with different body shapes and often result in "jerky" movements that saturate motors. DeepWalk addresses these issues with a "minimal craftsmanship" approach: a single policy that learns to walk in any direction purely through interaction, without needing a library of human movements to copy.
Methods
The researchers utilized Proximal Policy Optimization (PPO) with Generalized Advantage Estimation (GAE) to train the agent in the MuJoCo physics simulator. The system uses a single neural network to map the robot's state directly to joint actions.
Key technical innovations include:
- Beta Policies: Instead of the standard Gaussian distribution for actions, they used Beta distributions. This naturally bounds the action space (joint increments), preventing the policy from requesting impossible torques that lead to hardware damage.
- Nominal Pose Regularization: To avoid the need for motion capture data, they used a simple "standing pose" as a regularization target. This encourages the robot to stay near a natural configuration without forcing it to follow a specific gait cycle.
- Velocity Scheduler (Curriculum Learning): A scheduler gradually increases the range of target velocities ($v_x, v_y, \omega_z$) as training progresses. This prevents the robot from simply learning to stand still or slide.
- Sim-to-Real Pipeline: To bridge the "reality gap," they implemented dynamics randomization, sensor noise injection, and actuation latency modeling to account for the delay between a command and the robot's physical response.
Key Results
The framework was validated on the NimbRo-OP2X, a 135cm-tall humanoid robot. The single policy proved capable of handling forward, backward, lateral, and turning motions simultaneously, demonstrating the power of information transfer between different walking modes.
Quantitative and qualitative results include:
- Omnidirectional Mastery: The robot learned to walk at various speeds in all directions, including diagonal movement and turning while walking.
- Robustness: In simulation, the controller achieved a 93.75% success rate when switching between 160 different random velocity commands. On physical hardware, the robot could withstand significant pushes (up to 40-60N while standing and 30-50N while walking) without falling.
- Successful Transfer: The learned policy was successfully deployed on the NimbRo-OP2X hardware, demonstrating a stable gait that matched simulated behavior despite real-world friction and joint backlash.
Conclusion
DeepWalk demonstrates that a single neural network can achieve robust, omnidirectional bipedal locomotion without the need for complex reference motions. By addressing hardware constraints through Beta policies and latency modeling, the authors achieved a reliable sim-to-real transfer. While currently limited to flat ground, future work will explore extending the framework to complex terrains like stairs and uneven surfaces.
Summary updated