Abstract
Bridging the "reality gap" between simulation and physical hardware remains a significant challenge in robotic locomotion. This paper proposes a hybrid framework that utilizes Deep Reinforcement Learning (D-RL) to discover optimal gait patterns in simulation, followed by the extraction of low-dimensional Kinematic Motion Primitives (kMPs) via Principal Component Analysis (PCA). This approach allows for the deployment of robust, computationally efficient gaits—including trotting, galloping, and bounding—onto a custom quadruped robot, "Stoch," without the need for on-board execution of deep neural networks.
Introduction
While Deep Reinforcement Learning (D-RL) has proven effective in teaching robots complex locomotion in simulated environments, direct transfer to real-world hardware is often hindered by the "reality gap." Discrepancies in motor friction, sensor noise, and structural compliance frequently cause simulated policies to fail in practice. Furthermore, the high computational cost of running deep policies on small, embedded controllers limits real-time performance. This research addresses these challenges by distilling high-dimensional joint trajectories into a few fundamental waveforms, or kMPs, which are both computationally light and transferable across different gaits.
Methods
The proposed framework consists of a three-stage pipeline: learning, extraction, and deployment.
- Learning in Simulation: Proximal Policy Optimization (PPO) is employed within the Pybullet simulator. To enhance learning efficiency, the agent controls foot end-points in polar coordinates rather than raw joint angles, using inverse kinematics for translation.
- Extracting Primitives (kMPs): Stable joint trajectories recorded from simulation are analyzed using PCA. This process identifies the underlying low-dimensional structure of the movement.
- Hardware Deployment: Instead of the original neural network, the robot uses the extracted kMPs to reconstruct joint trajectories. A "synergy map" relates these primitives to specific joints, allowing for the generation of various gaits by adjusting phase relationships.
Results
The experimental results highlight the efficiency of the kMP approach. Analysis showed that walking is fundamentally a low-dimensional problem; the first principal component alone captured 88.9% of the motion variance, while the first four components accounted for 99.9%.
The "Stoch" robot successfully demonstrated four distinct gaits (trot, walk, gallop, and bound) in real-world tests. The robot achieved peak speeds of 0.60 m/s for the trot and 0.55 m/s for the bound. Notably, the kMPs extracted from the RL agent closely mirrored the motion primitives observed in biological systems, such as horses, suggesting that D-RL naturally converges toward physics-optimal solutions found in nature.
Conclusion
This work demonstrates that complex, learned locomotion can be effectively realized on physical hardware through the use of Kinematic Motion Primitives. By reducing the dimensionality of the control problem, the authors achieved high-performance locomotion with low computational overhead. Future research will focus on automating the mapping of synergy matrices and representing kMPs as continuous mathematical functions to further enhance robustness and scalability.
Summary updated