Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives

Abhik Singla, Shounak Bhattacharya, Dhaivat Dholakiya, Shalabh Bhatnagar, Ashitava Ghosal, Bharadwaj Amrutur, Shishir Kolathaya

This paper, from researchers at the Indian Institute of Science (IISc), presents a hybrid approach to quadrupedal locomotion that combines the power of Deep Reinforcement Learning (D-RL) with the structural efficiency of Kinematic Motion Primitives (kMPs). By extracting the "essence" of a learned gait in simulation, the authors demonstrate a robust way to deploy multiple walking behaviors on real hardware with minimal computational overhead.

TL;DR

The authors use Deep Reinforcement Learning to "learn" walking in simulation, then use Principal Component Analysis (PCA) to compress these complex trajectories into a few basic building blocks called Kinematic Motion Primitives (kMPs). These primitives allow a custom quadruped robot, "Stoch," to perform multiple gaits—trot, walk, gallop, and bound—on real hardware without needing to retrain the neural network for each specific behavior.

What Problem Does This Paper Address?

Developing controllers for legged robots is notoriously difficult. Traditionally, researchers chose between two paths:

  1. Model-Based Control: Using complex physics equations (like inverted pendulums). These are precise but struggle with the "reality gap"—the difference between a perfect math model and a messy, physical robot.
  2. Deep Reinforcement Learning (D-RL): Letting a robot learn through trial and error in simulation. While D-RL produces robust behaviors, the resulting policies are often "black boxes" that are computationally expensive to run on-board (e.g., on a Raspberry Pi) and require massive retraining for every new gait or hardware tweak.

The authors address this by using D-RL as a "trajectory generator" rather than a direct controller. They seek a middle ground that is data-driven yet structured and transferable.

Methods

The workflow consists of three main stages:

  1. Learning in Simulation: The authors trained a policy using Proximal Policy Optimization (PPO) in the Pybullet simulator. To improve efficiency, they chose an action space based on leg end-point positions in polar coordinates rather than raw joint angles, which helped avoid self-collisions.
  2. Extracting kMPs: Once a stable gait (like a trot) was learned, they recorded the joint trajectories and applied Principal Component Analysis (PCA). PCA identifies the most significant patterns in the data. They found that just four principal components (the kMPs) could reconstruct the original motion with over 99.9% accuracy.
  3. Hardware Deployment: Instead of running the full neural network on the robot, they used the extracted kMPs. By applying a "synergy matrix"—which defines the timing and phase shifts between legs—they could reconstruct joint trajectories for the real robot.

Datasets

  • Simulation Data: Generated using 30 parallel agents in the Pybullet simulator.
  • Biological Comparison: The authors compared their robot's learned kMPs against existing datasets of horse locomotion.
  • Experimental Data: Recorded from "Stoch," their custom-built quadruped featuring 8 actuators (2 per leg) and a Raspberry Pi 3-B as the primary compute module.

Key Insights and Ideas

  • The "VNMW" Pattern: The authors observed that kMPs follow a distinct visual pattern (resembling the letters V, N, M, and W). Remarkably, this pattern was consistent across different gaits and even matched kMPs extracted from biological horses. This suggests that there is a fundamental, low-dimensional structure to efficient quadrupedal walking.
  • One Training, Many Gaits: Perhaps the most significant insight is that you don't need to retrain a D-RL agent to get a new gait. By manually adjusting the "synergy matrix" (the phase relationship between the legs), the authors derived walking, galloping, and bounding from a single learned trotting gait.
  • Computational Efficiency: By using kMPs, the robot only needs to track a few pre-calculated waveforms rather than performing heavy neural network inference, making it ideal for low-power embedded hardware.

Key Results

  • High Fidelity: The reconstruction of joint trajectories from just 4 kMPs showed a cross-covariance of over 0.90 when compared to the original simulation data.
  • Speed: The robot achieved a maximum walking speed of 0.60 m/s during a trot.
  • Versatility: The team successfully demonstrated four distinct gaits (trot, walk, gallop, bound) on the physical Stoch robot, as well as robustness to external pushes.

Limitations and Future Work

  • Manual Synergy Tuning: Currently, the "synergy matrices" used to derive new gaits from the kMPs are tuned manually. The authors suggest that future work could involve using optimization algorithms to find these matrices automatically.
  • Environment Adaptation: While kMPs are robust, the current system is largely open-loop regarding the gait pattern itself. Future iterations could involve "closing the loop" by allowing the kMPs to adapt dynamically to rough terrain in real-time.
  • Continuous Primitives: The authors aim to move from discrete data-point kMPs to continuous mathematical functions (like Bezier polynomials), which would further simplify the control architecture.

Summary updated

Abstract

Humans and animals are believed to use a very minimal set of trajectories to perform a wide variety of tasks including walking. Our main objective in this paper is two fold 1) Obtain an effective tool to realize these basic motion patterns for quadrupedal walking, called the kinematic motion primitives (kMPs), via trajectories learned from deep reinforcement learning (D-RL) and 2) Realize a set of behaviors, namely trot, walk, gallop and bound from these kinematic motion primitives in our custom four legged robot, called the “Stoch”. D-RL is a data driven approach, which has been shown to be very effective for realizing all kinds of robust locomotion behaviors, both in simulation and in experiment. On the other hand, kMPs are known to capture the underlying structure of walking and yield a set of derived behaviors. We first generate walking gaits from D-RL, which uses policy gradient based approaches. We then analyze the resulting walking by using principal component analysis. We observe that the kMPs extracted from PCA followed a similar pattern irrespective of the type of gaits generated. Leveraging on this underlying structure, we then realize walking in Stoch by a straightforward reconstruction of joint trajectories from kMPs. This type of methodology improves the transferability of these gaits to real hardware, lowers the computational overhead on-board, and also avoids multiple training iterations by generating a set of derived behaviors from a single learned gait.
BibTeX
@inproceedings{singla2019realizing,
  author = {Abhik Singla and Shounak Bhattacharya and Dhaivat Dholakiya and Shalabh Bhatnagar and Ashitava Ghosal and Bharadwaj Amrutur and Shishir Kolathaya},
  title = {{Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=QegR1WWeSf},
}
Figure