DeepWalk: Omnidirectional Bipedal Gait by Deep Reinforcement Learning

Diego Rodriguez, Sven Behnke

TL;DR

Researchers have developed DeepWalk, a Deep Reinforcement Learning (DRL) framework that enables humanoid robots to achieve robust, omnidirectional walking using a single neural network policy. Unlike many previous approaches, it requires no reference motion data (like motion capture) and successfully transfers from simulation to a 135cm-tall real-world humanoid robot.

What Problem Does This Paper Address?

Bipedal walking is a "grand challenge" in robotics due to high-dimensional dynamics, unstable equilibrium, and strict real-time constraints. Traditionally, engineers used complex mathematical models or "stitched together" multiple controllers for different speeds and directions.

While DRL has shown promise, most existing methods rely on reference motions (mimicking human data) or engineered modules (like trajectory planners). These dependencies make it hard to adapt the gait to robots with different body shapes and often result in "jerky" movements that saturate motors. The authors sought a "minimal craftsmanship" approach: a single policy that learns to walk in any direction purely through interaction, without needing a library of human movements to copy.

Methods

The authors used Proximal Policy Optimization (PPO) with Generalized Advantage Estimation (GAE) to train the agent in the MuJoCo physics simulator. Key technical innovations include:

  • Beta Policies: Instead of the standard Gaussian (Normal) distribution for actions, they used Beta distributions. This naturally bounds the action space (joint increments), preventing the policy from requesting impossible torques that lead to "jerky" motions and hardware damage.
  • Nominal Pose Regularization: To avoid the need for motion capture data, they used a simple "standing pose" as a regularization target. This encourages the robot to stay near a natural configuration without forcing it to follow a specific gait cycle.
  • Velocity Scheduler (Curriculum Learning): The robot isn't asked to run immediately. A scheduler gradually increases the range of target velocities ($v_x, v_y, \omega_z$) as training progresses. They introduced a "core velocity" to prevent the robot from simply learning to stand still or slide.
  • Sim-to-Real Pipeline: To bridge the "reality gap," they implemented dynamics randomization (friction, masses), sensor noise injection, and—crucially—actuation latency modeling to account for the delay between a command and the robot's physical response.

Datasets

As a robotics control paper, it does not use a static dataset. Instead, it utilizes:

  • MuJoCo Physics Engine: For generating millions of training samples.
  • NimbRo-OP2X Model: A digital twin of their 135cm, 18-degree-of-freedom humanoid robot.
  • Real-world Evaluation: Testing on the physical NimbRo-OP2X hardware to validate the sim-to-real transfer.

Key Insights and Ideas

  • Single Policy Supremacy: The paper proves that a single neural network can handle forward, backward, lateral, and turning motions simultaneously. This facilitates "information transfer" between different walking modes, making the overall system more cohesive than a collection of specialized controllers.
  • The Power of the "Nominal Pose": You don't need complex human animations to teach a robot to walk. A simple mathematical pull toward a standing pose is enough to guide the RL agent toward a stable, anthropomorphic gait.
  • Latency is the Enemy: One of the biggest hurdles in sim-to-real is time. By explicitly modeling the history of observations and the delay in motor response, the researchers made the policy robust enough to handle real-world hardware lag.
  • Beta > Gaussian: For physical hardware, bounding actions at the policy level (via Beta distributions) is more effective than "clipping" Gaussian outputs, leading to smoother movements and faster convergence.

Key Results

  • Omnidirectional Mastery: The robot learned to walk at various speeds in all directions, including diagonal movement and turning while walking.
  • Robustness: In simulation, the controller achieved a 93.75% success rate when switching between 160 different random velocity commands.
  • Perturbation Resistance: The real robot could withstand significant pushes (up to 40-60N while standing and 30-50N while walking) without falling.
  • Successful Transfer: The learned policy was successfully deployed on the NimbRo-OP2X hardware, demonstrating a stable gait that matched the simulated behavior despite the complexities of real-world friction and joint backlash.

Limitations and Future Work

While DeepWalk is highly effective on flat ground, the authors acknowledge several areas for growth:

  • Terrain Complexity: The current model was trained and tested on flat surfaces. It does not yet handle stairs, slopes, or uneven "stepping stone" environments.
  • Gait Aesthetics: While functional and stable, the gait is learned from scratch and may not always look as "human-like" as methods that use motion capture.
  • Future Directions: The team plans to explore more flexible motion regularizers and extend the framework to 3D navigation tasks, such as climbing stairs and navigating complex obstacles.

Summary updated

Abstract

Bipedal walking is one of the most difficult but exciting challenges in robotics. The difficulties arise from the complexity of high-dimensional dynamics, sensing and actuation limitations combined with real-time and computational constraints. Deep Reinforcement Learning (DRL) holds the promise to address these issues by fully exploiting the robot dynamics with minimal craftsmanship. In this paper, we propose a novel DRL approach that enables an agent to learn omnidirectional locomotion for humanoid (bipedal) robots. Notably, the locomotion behaviors are accomplished by a single control policy (a single neural network). We achieve this by introducing a new curriculum learning method that gradually increases the task difficulty by scheduling target velocities. In addition, our method does not require reference motions which facilities its application to robots with different kinematics, and reduces the overall complexity. Finally, different strategies for sim-to-real transfer are presented which allow us to transfer the learned policy to a real humanoid robot.
BibTeX
@inproceedings{rodriguez2021deepwalk,
  author = {Diego Rodriguez and Sven Behnke},
  title = {{DeepWalk: Omnidirectional Bipedal Gait by Deep Reinforcement Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=vrlx9TJFp-1},
}

Loading related content...

Figure