Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loads

Jeremy Dao, Kevin Green, Helei Duan, Alan Fern, Jonathan W. Hurst

Introduction

While bipedal robots have demonstrated remarkable agility in controlled environments, their utility in real-world applications depends on their ability to transport varying payloads. Most current research focuses on "unloaded" locomotion, where the robot's dynamics are relatively predictable. However, practical tasks often involve carrying "dynamic" loads—objects like sloshing liquids, swinging weights, or pulled carts—that exert unpredictable, time-varying forces on the robot's frame.

Traditionally, managing these forces requires specialized hardware, such as force-torque sensors, or complex mathematical models of the load's dynamics. This research explores an alternative: training the bipedal robot Cassie to handle unsensed dynamic loads using only its standard internal sensors (proprioception) through Deep Reinforcement Learning (RL).

Methodology

Architecture and Training

The researchers utilized the bipedal robot Cassie as their experimental platform. To process the temporal nature of dynamic loads, they employed a Long Short-Term Memory (LSTM) neural network architecture. The LSTM's ability to maintain internal state allows the robot to "feel" the rhythm and momentum of shifting loads over time, effectively inferring the load's properties from proprioceptive feedback alone.

The policies were trained in the MuJoCo physics simulator using Proximal Policy Optimization (PPO). To bridge the "sim-to-real gap," the team applied Dynamics Randomization, varying physical parameters like joint friction and mass during training to ensure the policy remained robust when transferred to physical hardware.

The Bootstrapping Technique

A key innovation in this work is the use of "bootstrapping." Rather than training each load-specific policy from scratch, the researchers began with a pre-trained policy optimized for unloaded walking. This baseline was then fine-tuned for specific dynamic loads. This approach proved highly efficient, saving approximately 100 million samples of training data and resulting in more stable performance compared to policies trained from a cold start.

Experimental Scenarios

The study evaluated the robot's performance across five distinct simulation environments, each representing a unique dynamic challenge:

  1. Unloaded Cassie: The baseline for comparison.
  2. Tray Box: A 5kg box placed freely on a tray, requiring the robot to maintain a level pelvis to prevent the box from sliding off.
  3. Cassie Cart: Pulling a cart via ropes, which introduces persistent backward-pulling forces.
  4. Carrying Pole: Two 5kg masses suspended from a pole, creating significant lateral and torsional swinging forces.
  5. Water Jug: A box containing an internal spring-mass system designed to approximate the complex, non-linear forces of sloshing liquid.

Key Results and Insights

Emergent Gait Strategies

One of the most significant findings was the emergence of specialized gait strategies tailored to specific loads. The robot did not simply apply more force; it fundamentally altered its walking pattern. For the Tray Box task, Cassie adopted a narrower stance to stabilize its pelvis. Conversely, for the Carrying Pole task, the robot learned to take wider steps, creating a broader base of support to counteract the lateral momentum of the swinging weights.

Performance and Robustness

The learned policies showed dramatic improvements over baseline controllers. In the "Carrying Pole" scenario—the most difficult task—the standard unloaded policy failed almost immediately (0% success rate). In contrast, the specialized learned policy achieved a success rate of over 95%. Furthermore, robots trained with dynamic loads demonstrated increased external robustness, resisting 10-20N more force from external pushes than the unloaded baseline.

Interestingly, the researchers also developed a "General Load Policy" trained on all scenarios simultaneously. This generalist policy performed nearly as well as the specialized ones, suggesting that the robot can learn a universal set of "robustness skills" applicable to various unsensed forces.

Sim-to-Real Transfer

The policies successfully transferred to the physical Cassie robot, enabling it to carry the 10kg swinging pole and the tray box in real-world conditions. However, a significant performance gap was observed regarding speed. While the Carrying Pole policy reached 2.5 m/s in simulation, it was limited to 0.8 m/s on hardware. This suggests that while the simulation captures the primary dynamics, it may not fully account for high-frequency vibrations and impacts present in the physical system.

Conclusion and Future Work

This research demonstrates that bipedal robots can autonomously adapt to complex, unsensed dynamic loads through RL and proprioception. By leveraging bootstrapping and LSTM architectures, the researchers enabled Cassie to navigate challenging transport tasks without specialized sensors. Future work will focus on narrowing the sim-to-real speed gap, potentially through real-time system identification or more sophisticated reward shaping to better model high-frequency physical interactions.

Summary updated

BibTeX
@inproceedings{dao2022sim,
  author = {Jeremy Dao and Kevin Green and Helei Duan and Alan Fern and Jonathan W. Hurst},
  title = {{Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loads}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  doi = {10.1109/icra46639.2022.9811783},
}
Figure