Zero-Shot Retargeting of Learned Quadruped Locomotion Policies Using Hybrid Kinodynamic Model Predictive Control

Wenhao Yu, He Li, Wenhao Yu 0003, Tingnan Zhang, Patrick M. Wensing

TL;DR

This paper presents a novel framework for "zero-shot" retargeting of learned quadruped locomotion policies to new robots without retraining. By combining Reinforcement Learning (RL) for generating dynamic trajectories and a new Hybrid Kinodynamic Model Predictive Control (HKD-MPC) for robust execution, they demonstrate successful transfer of complex gaits between robots of different sizes and dynamics. This significantly reduces the effort required to deploy learned skills on new robotic platforms.

What Problem Does This Paper Address?

Developing robust locomotion for legged robots is a significant challenge, often tackled by Reinforcement Learning (RL) or Model Predictive Control (MPC). While RL excels at discovering complex, dynamic behaviors, its policies are highly sensitive to the robot's dynamics, making direct transfer to a different robot (even with the same topology) difficult and usually requiring extensive retraining. This "sim-to-real" gap and the need for retraining are major hurdles for deploying learned skills on new hardware. Existing MPC approaches, while good at stabilization, often struggle to generate complex, dynamic motions from scratch or require good initial trajectories, which can be hard to obtain. This paper aims to bridge this gap by enabling the reuse of pre-trained RL locomotion policies on new robots without any retraining, effectively achieving zero-shot transfer.

Methods

The core of the proposed framework is a two-stage pipeline:

  1. RL Policy Roll-out for Reference Generation: An existing RL policy, trained on a source robot (e.g., Unitree A1 or Laikago), is "unrolled" offline. This process generates a dynamically plausible trajectory for the robot's Center of Mass (CoM), joint positions, and importantly, a contact schedule (which feet are on the ground). This serves as a reference for the target robot.
  2. Hybrid Kinodynamic Model Predictive Control (HKD-MPC) for Retargeting: This generated reference trajectory and contact schedule are then fed into a novel HKD-MPC controller designed for the target robot (e.g., MIT Mini Cheetah). The HKD-MPC uses a simplified model that captures the rigid body dynamics of the robot's base and the kinematics of the legs. Crucially, it simultaneously optimizes:
  3. Foothold locations: Where the feet should land.
  4. Ground Reaction Forces (GRFs): The forces exerted by the feet on the ground.
  5. CoM trajectory: To follow the RL reference as closely as possible while respecting the target robot's dynamics and constraints.

    The HKD-MPC's key innovation is its simplified model that only enforces foot position constraints at the moment of touchdown, rather than requiring complex, velocity-level non-slipping constraints during stance. This makes the optimization problem easier to solve while still ensuring stable locomotion. The MPC then generates the actual control commands for the target robot.

Datasets

The paper does not introduce new datasets. Instead, it leverages existing RL policies trained for quadruped locomotion, specifically mentioning those trained for the Unitree A1 and Laikago robots using the motion imitation learning framework from [10]. These policies were trained using physics simulators like PyBullet. The evaluation is performed on the MIT Mini Cheetah robot, both in simulation and on hardware.

Key Insights and Ideas

  • RL as a "Motion Generator," MPC as a "Robust Executor": The paper elegantly combines the strengths of both paradigms. RL is used for its ability to discover complex, dynamic motions, while MPC is used for its robustness, ability to handle constraints, and adapt to the specific dynamics of a new robot.
  • Zero-Shot Transfer is Achievable: The most significant insight is that complex locomotion skills can be transferred between robots with vastly different sizes and dynamics without any retraining of the RL policy. This is a major step towards reusability of learned robotic skills.
  • Simplified Dynamics for MPC is Sufficient: The HKD model, by simplifying the constraints and focusing on touchdown foot placement, proves effective for retargeting. This highlights that a perfect model isn't always necessary for robust control when provided with a good reference.
  • MPC Provides a "Safe Seed": The RL-generated trajectory acts as a "good seed" for the MPC, guiding it towards dynamic behaviors while the MPC ensures feasibility and stability on the target robot. This avoids the need for manual trajectory design or extensive MPC tuning for complex motions.

Key Results

  • Successful Retargeting: The framework successfully retargeted trotting, pacing, and turning policies trained on the Unitree A1 and Laikago robots to the MIT Mini Cheetah robot.
  • Hardware Validation: The retargeting was demonstrated not only in simulation but also on the physical Mini Cheetah robot, showcasing real-world applicability.
  • Comparison to Direct RL Deployment: Directly executing the RL policy trained on one robot (A1) on another (Mini Cheetah) resulted in immediate failure (falling down), underscoring the necessity of the proposed retargeting framework.
  • Qualitative Performance: Figures and accompanying videos show that the retargeted motions closely resemble the original RL policy roll-outs, demonstrating good tracking performance. Minor variations were observed, attributed to the PD swing controller and foot regularization.
  • Efficiency: The HKD-MPC runs at 50 Hz, with planning taking approximately 6 ms on a laptop, making it suitable for real-time control.

Limitations and Future Work

  • Sliding Contacts: The current HKD-MPC formulation assumes stationary contacts. If the RL policy roll-out involves sliding contacts, the robot may fall. The authors suggest future work could involve designing rewards to penalize slips or developing MPC that can handle sliding contacts.
  • Swing Controller Collision Avoidance: The swing controller interpolates foot trajectories but is not explicitly obstacle-aware. Future work could incorporate obstacle clearance constraints into the MPC and develop a collision-free swing controller.
  • Generalization to Diverse Morphologies: While successful for robots with similar topologies but different sizes, the paper notes that more general strategies (e.g., anisotropic scaling) might be needed for robots with significantly different morphologies.
  • Self-Collision: The simulation results did not account for self-collision, and a turning motion failed on hardware due to leg collision. Developing MPC that can handle self-collision is an area for future research.

Summary updated

Abstract

Reinforcement Learning (RL) has witnessed great strides for quadruped locomotion, with continued progress in the reliable sim-to-real transfer of policies. However, it remains a challenge to reuse a policy on another robot, which could save time for retraining. In this work, we present a framework for zero-shot policy retargeting wherein diverse motor skills can be transferred between robots of different shapes and sizes. The new framework centers on a planning-and-control pipeline that systematically integrates RL and Model Predictive Control (MPC). The planning stage employs RL to generate a dynamically plausible trajectory as well as the contact schedule, avoiding the combinatorial complexity of contact sequence optimization. This information is then used to seed the MPC to stabilize and robustify the policy roll-out via a new Hybrid Kinodynamic (HKD) model that implicitly optimizes the foothold locations. Hardware results show an ability to transfer policies from both the A1 and Laikago robots to the MIT Mini Cheetah robot without requiring any policy re-tuning.
BibTeX
@inproceedings{yu2022zero,
  author = {Wenhao Yu and He Li and Wenhao Yu and Tingnan Zhang and Patrick M. Wensing},
  title = {{Zero-Shot Retargeting of Learned Quadruped Locomotion Policies Using Hybrid Kinodynamic Model Predictive Control}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981967},
}
Figure