Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives

Abhik Singla, Shounak Bhattacharya, Dhaivat Dholakiya, Shalabh Bhatnagar, Ashitava Ghosal, Bharadwaj Amrutur, Shishir Kolathaya

Abstract

Bridging the "reality gap" between simulation and physical hardware remains a significant challenge in robotic locomotion. This paper proposes a hybrid framework that utilizes Deep Reinforcement Learning (D-RL) to discover optimal gait patterns in simulation, followed by the extraction of low-dimensional Kinematic Motion Primitives (kMPs) via Principal Component Analysis (PCA). This approach allows for the deployment of robust, computationally efficient gaits—including trotting, galloping, and bounding—onto a custom quadruped robot, "Stoch," without the need for on-board execution of deep neural networks.

Introduction

While Deep Reinforcement Learning (D-RL) has proven effective in teaching robots complex locomotion in simulated environments, direct transfer to real-world hardware is often hindered by the "reality gap." Discrepancies in motor friction, sensor noise, and structural compliance frequently cause simulated policies to fail in practice. Furthermore, the high computational cost of running deep policies on small, embedded controllers limits real-time performance. This research addresses these challenges by distilling high-dimensional joint trajectories into a few fundamental waveforms, or kMPs, which are both computationally light and transferable across different gaits.

Methods

The proposed framework consists of a three-stage pipeline: learning, extraction, and deployment.

  1. Learning in Simulation: Proximal Policy Optimization (PPO) is employed within the Pybullet simulator. To enhance learning efficiency, the agent controls foot end-points in polar coordinates rather than raw joint angles, using inverse kinematics for translation.
  2. Extracting Primitives (kMPs): Stable joint trajectories recorded from simulation are analyzed using PCA. This process identifies the underlying low-dimensional structure of the movement.
  3. Hardware Deployment: Instead of the original neural network, the robot uses the extracted kMPs to reconstruct joint trajectories. A "synergy map" relates these primitives to specific joints, allowing for the generation of various gaits by adjusting phase relationships.
The three-stage pipeline for realizing learned locomotion: Learning in simulation, Data Collection for kMP extraction, and Hardware Deployment.

Results

The experimental results highlight the efficiency of the kMP approach. Analysis showed that walking is fundamentally a low-dimensional problem; the first principal component alone captured 88.9% of the motion variance, while the first four components accounted for 99.9%.

Reconstruction of joint trajectories using kMPs. The table shows that four primitives capture 99.9% of the motion variance.

The "Stoch" robot successfully demonstrated four distinct gaits (trot, walk, gallop, and bound) in real-world tests. The robot achieved peak speeds of 0.60 m/s for the trot and 0.55 m/s for the bound. Notably, the kMPs extracted from the RL agent closely mirrored the motion primitives observed in biological systems, such as horses, suggesting that D-RL naturally converges toward physics-optimal solutions found in nature.

Conclusion

This work demonstrates that complex, learned locomotion can be effectively realized on physical hardware through the use of Kinematic Motion Primitives. By reducing the dimensionality of the control problem, the authors achieved high-performance locomotion with low computational overhead. Future research will focus on automating the mapping of synergy matrices and representing kMPs as continuous mathematical functions to further enhance robustness and scalability.

Summary updated

Abstract

Humans and animals are believed to use a very minimal set of trajectories to perform a wide variety of tasks including walking. Our main objective in this paper is two fold 1) Obtain an effective tool to realize these basic motion patterns for quadrupedal walking, called the kinematic motion primitives (kMPs), via trajectories learned from deep reinforcement learning (D-RL) and 2) Realize a set of behaviors, namely trot, walk, gallop and bound from these kinematic motion primitives in our custom four legged robot, called the “Stoch”. D-RL is a data driven approach, which has been shown to be very effective for realizing all kinds of robust locomotion behaviors, both in simulation and in experiment. On the other hand, kMPs are known to capture the underlying structure of walking and yield a set of derived behaviors. We first generate walking gaits from D-RL, which uses policy gradient based approaches. We then analyze the resulting walking by using principal component analysis. We observe that the kMPs extracted from PCA followed a similar pattern irrespective of the type of gaits generated. Leveraging on this underlying structure, we then realize walking in Stoch by a straightforward reconstruction of joint trajectories from kMPs. This type of methodology improves the transferability of these gaits to real hardware, lowers the computational overhead on-board, and also avoids multiple training iterations by generating a set of derived behaviors from a single learned gait.
BibTeX
@inproceedings{singla2019realizing,
  author = {Abhik Singla and Shounak Bhattacharya and Dhaivat Dholakiya and Shalabh Bhatnagar and Ashitava Ghosal and Bharadwaj Amrutur and Shishir Kolathaya},
  title = {{Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=QegR1WWeSf},
}
Figure