Abstract
We present a sim-to-real framework that uses dynamics and domain randomized offline reinforcement learning to enhance open-loop gaits for legged robots, allowing them to traverse uneven terrain without sensing foot impacts. Our approach, D$^2$-Randomized Gait Modulation with Bezier Curves (D$^2$-GMBC), uses augmented random search with randomized dynamics and terrain to train, in simulation, a policy that modifies the parameters and output of an open-loop Bezier curve gait generator for quadrupedal robots. The policy, using only inertial measurements, enables the robot to traverse unknown rough terrain, even when the robot's physical parameters do not match the open-loop model. We compare the resulting policy to hand-tuned Bezier Curve gaits and to policies trained without randomization, both in simulation and on a real quadrupedal robot. With D$^2$-GMBC, across a variety of experiments on unobserved and unknown uneven terrain, the robot walks significantly farther than with either hand-tuned gaits or gaits learned without domain randomization. Additionally, using D$^2$-GMBC, the robot can walk laterally and rotate while on the rough terrain, even though it was trained only for forward walking.
Introduction
Legged robots are theoretically ideal for navigating complex, uneven environments like disaster zones or construction sites. However, real-world rough terrain is difficult to model accurately. Most current solutions rely on precise foot-impact sensing or complex vision systems, which are expensive and prone to failure in dusty or low-light conditions. Furthermore, the "reality gap" in robotics—where a policy performing perfectly in simulation fails on a real robot due to unmodeled mass, friction, or terrain geometry—remains a significant hurdle.
This paper introduces a framework designed to create gaits inherently robust to these variations using only basic onboard sensors, specifically an Inertial Measurement Unit (IMU). By combining traditional control methods with randomized reinforcement learning, we achieve robust sim-to-real transfer on low-cost hardware.
Methodology: $D^2$-GMBC
The proposed framework, Dynamics and Domain Randomized Gait Modulation with Bezier Curves ($D^2$-GMBC), utilizes a hybrid architecture to bridge the reality gap.
System Architecture
The system consists of two primary components:
- Base Gait Generator: An open-loop Bezier curve generator provides a stable, rhythmic baseline movement.
- Modulation Policy ($\pi$): A simple linear policy trained using Augmented Random Search (ARS). Instead of replacing the gait, this policy modulates it by taking IMU data (roll, pitch, angular velocity) and outputting small adjustments to foot positions and gait parameters like step height.
Double Randomization ($D^2$)
The "secret sauce" of this approach is the extreme randomization during training in the PyBullet physics engine. We randomize both:
- Dynamics: The mass of the robot's limbs and the friction of its feet.
- Domain: The geometry of the ground, creating a mesh of varying heights to simulate unpredictable rocky terrain.
Because the resulting policy is linear, it is computationally efficient enough to run on basic hobbyist-grade microcontrollers found on low-cost robots.
Experimental Evaluation
We evaluated $D^2$-GMBC against hand-tuned Bezier gaits and policies trained without terrain randomization, using both simulation and a real-world OpenQuadruped robot.
Key Results
The results demonstrate a significant leap in reliability over standard methods:
- Simulated Survival: In 1,000 trials over rough terrain, $D^2$-GMBC achieved a 45% survival rate for long-distance travel (>90m). Policies trained without terrain randomization survived only 8% of the time, and open-loop gaits failed 100% of the time.
- Real-World Success: On a physical "Rocky Test Track," $D^2$-GMBC showed a 4.28x improvement in success rate compared to standard hand-tuned Bezier gaits. The robot traveled roughly 40% farther on average using the randomized policy.
Discussion and Conclusion
Our findings suggest that while policies trained without terrain randomization achieve higher rewards during training, they fail to generalize to real-world irregularities. The $D^2$-randomized policy, though harder to train, developed emergent robustness—allowing the robot to walk laterally and rotate on rough terrain despite being trained only for forward walking.
While successful, hardware limitations like servo torque limits remain a bottleneck for more dynamic maneuvers. Future work will focus on applying this framework to torque-controllable systems to enable even greater mobility in complex environments.
Summary updated