Learning Dynamic Bipedal Walking Across Stepping Stones

Helei Duan, Ashish Malik, Mohitvishnu S. Gadde, Jeremy Dao, Alan Fern, Jonathan W. Hurst

Introduction

Bipedal robots are theoretically well-suited for human environments, yet they frequently struggle with "discrete terrains"—environments such as rocky paths or stepping stones where foot placement is a strict requirement for stability rather than a suggestion. Navigating these terrains requires a robot to reason about its complex internal dynamics and the external environment simultaneously, often at high speeds.

Existing approaches to this problem typically fall into two categories: model-based optimizations, which are often too computationally expensive for real-time execution, and reinforcement learning (RL) methods, which frequently lack the closed-loop visual feedback necessary to adjust to real-world variations. This research introduces a novel RL framework that enables the bipedal robot Cassie to dynamically traverse stepping stones in real-time by combining a modular control architecture with a reachability prediction model.

Methodology

The proposed system consists of two primary components: a modular Control Policy and a Reachability Prediction Model.

Modular Control Policy

Rather than training a monolithic controller from scratch, the researchers utilized a modular neural network designed for efficiency and transferability:

  • LSTM Dynamics Module: This module processes proprioceptive data (internal sensors such as joint positions and velocities). By bootstrapping this module from a pre-trained general walking policy, the team significantly reduced training time.
  • Feed-Forward (FF) Module: This component integrates the dynamics module's output with a periodic clock and specific footstep commands.
  • Variable Timing: A critical innovation is the policy's ability to output both motor commands and a "clock increment." This allows the robot to dynamically adjust its gait frequency—speeding up or slowing down to ensure it hits a target—mimicking the natural adjustments found in human locomotion.

Reachability Prediction Model

To ensure safety and stability, a separate supervised learning model acts as a "reachability" filter. This model predicts the likely landing error and the robot's resulting state for any given footstep target. By acting as a sanity check, it allows a high-level planner to select targets that are physically reachable given the robot's current momentum, rather than simply aiming for the nearest stone.

Sim-to-Real Pipeline

The system was trained in simulation using Proximal Policy Optimization (PPO). To bridge the gap between simulation and reality, the researchers employed heavy dynamics randomization, varying factors like mass and friction. For physical hardware tests, an overhead camera tracked ArUco markers on a treadmill, providing Cassie with real-time target coordinates for closed-loop control.

Experimental Results

The framework was evaluated using a new "Stepping Stone Benchmark" consisting of eight patterns (A through H) that scale in difficulty based on step count, angle extremity, and the abruptness of transitions.

Simulation Performance

In simulated trials, the robot achieved high precision, with average footstep errors ranging between 7 and 11 cm, even on the most complex 8-step sequences. When tested in a "multiple choice" scenario—crossing a gap with ten randomly placed stones—the reachability model proved its worth. The robot achieved an 82.3% success rate using the reachability filter, compared to a 50.2% success rate when using a naive "closest stone" strategy.

Hardware Demonstration

The policy successfully transferred to the physical Cassie robot. In real-time treadmill tests, Cassie demonstrated the ability to traverse the benchmark patterns using closed-loop vision, marking a significant milestone in perception-driven bipedal locomotion.

Discussion and Future Work

While the results are promising, several limitations remain. The current implementation relies on an external overhead camera system. Transitioning to on-board vision is a necessary next step for true autonomy, though it introduces challenges related to camera shake and sensor occlusion during movement.

Additionally, the current policy only considers the immediate next step. Future iterations of the reachability model will aim to support multi-step look-ahead planning. This would allow the robot to navigate complex fields of stones by planning a trajectory that accounts for long-term momentum, rather than reacting step-by-step. Finally, addressing the robot's sensitivity to its initial state (speed and orientation) will be crucial for making the system more robust in unpredictable real-world environments.

Summary updated

BibTeX
@inproceedings{duan2022learning,
  author = {Helei Duan and Ashish Malik and Mohitvishnu S. Gadde and Jeremy Dao and Alan Fern and Jonathan W. Hurst},
  title = {{Learning Dynamic Bipedal Walking Across Stepping Stones}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981884},
}
Figure