RMP2: A Structured Composable Policy Class for Robot Learning

Anqi Li, Ching-An Cheng, Muhammad Asif Rana, Man Xie, Karl Van Wyk, Nathan D. Ratliff, Byron Boots

Abstract

Robot learning often faces a fundamental trade-off between the safety and stability of analytical control and the flexibility of machine learning. We present RMP$^2$, a practical implementation of the RMPflow framework designed for modern robot learning pipelines. By reformulating RMPflow's message-passing as a closed-form solution powered by automatic differentiation (Autodiff), RMP$^2$ allows for the seamless integration of control-theoretic priors into end-to-end reinforcement learning. Our results demonstrate that RMP$^2$ enables zero-collision exploration and significantly accelerates convergence in complex manipulation tasks.

Introduction

In robotics, researchers often choose between analytical control, which is safe but difficult to design for complex tasks, and machine learning, which is flexible but often unsafe and data-hungry. RMPflow was introduced to bridge this gap by combining multiple "Riemannian Motion Policies" (RMPs)—such as goal-reaching, collision avoidance, and joint-limit centering—into a single acceleration-based policy.

However, the original RMPflow implementation posed significant challenges for learning. It required manual definition of rigid tree structures and used a custom message-passing algorithm that was difficult to differentiate through. RMP$^2$ addresses these issues by re-imagining the framework through the lens of Automatic Differentiation, making it compatible with standard deep learning tools like PyTorch and TensorFlow.

The RMP$^2$ Framework

The key technical insight of RMP$^2$ is that the RMPflow message-passing algorithm is mathematically equivalent to solving a specific weighted least-squares optimization problem. Instead of manual "pushforward" and "pullback" steps, RMP$^2$ derives a closed-form solution for the robot's acceleration.

To compute this efficiently, the authors leverage three standard Autodiff primitives:

  1. gradient: For scalar outputs.
  2. jacobian: To compute task-map derivatives.
  3. jvp (Jacobian-vector product): To compute velocities and curvature terms without explicitly forming massive matrices.

By using double backpropagation, RMP$^2$ achieves the same $O(N)$ linear time complexity as the original RMPflow but with a simpler user interface and support for Directed Acyclic Graphs (DAGs), which is essential for complex robots with shared kinematic chains.

Experimental Setup

The researchers evaluated RMP$^2$ using two simulated environments in PyBullet:

  • Three-Link Planar Robot: A 2D arm tasked with reaching goals while avoiding cylindrical obstacles across three difficulty levels (varying goal range and obstacle density).
  • Franka Emika Panda: A 7-degree-of-freedom (DOF) industrial manipulator in a 3D workspace, tasked with reaching goals while avoiding spherical obstacles.
The experimental setup includes a 3-link planar robot (a) and a 7-DOF Franka Panda (b). Performance metrics (c, d) highlight the advantages of RMP-based residual learning in high-dimensional tasks.

Key Results and Discussion

Safety as a Prior

A standout feature of RMP$^2$ is the use of Residual RMP learning. Instead of training a neural network from scratch, the learner optimizes a "residual" policy on top of a hand-designed RMP prior. Because the structured policy class has built-in collision avoidance, the robot maintains safety even during the earliest, most random phases of reinforcement learning.

In the Franka robot task, the RMP-Residual policy maintained zero collisions throughout the entire training process. In contrast, standard neural network policies suffered from frequent crashes during exploration.

Safety percentage comparison across different environment complexities. RMP-RESIDUAL (blue) maintains near-perfect safety regardless of obstacle density or goal range, whereas standard NN policies (gray) struggle.

Convergence and Efficiency

RMP$^2$ significantly accelerates learning. In the most challenging 3-link arm environment, the RMP-Residual approach improved performance by more than 100% compared to standard neural networks within the same training budget. Furthermore, benchmarks showed that RMP$^2$ is significantly faster than naïve Autodiff implementations, especially as the complexity of the robot's task map increases.

Limitations and Future Work

While RMP$^2$ simplifies implementation, it currently focuses on acceleration-based (kinematic) control. Extending this to full torque-level dynamics for highly dynamic systems like quadrupeds remains an open challenge. Additionally, while the framework is safer, high-dimensional tasks like the 7-DOF Franka experiments suggest that structured policies may require further tuning of learning hyperparameters to reach optimal rewards within short training windows.

Conclusion

RMP$^2$ provides a practical, high-performance bridge between control theory and modern robot learning. By embedding safety and stability guarantees directly into the policy class, it enables more robust and efficient training for complex robotic tasks.

Summary updated

Abstract

We consider the problem of learning motion policies for acceleration-based robotics systems with a structured policy class specified by RMPflow. RMPflow is a multi-task control framework that has been successfully applied in many robotics problems. Using RMPflow as a structured policy class in learning has several benefits, such as sufficient expressiveness, the flexibility to inject different levels of prior knowledge as well as the ability to transfer policies between robots. However, implementing a system for end-to-end learning RMPflow policies faces several computational challenges. In this work, we re-examine the message passing algorithm of RMPflow and propose a more efficient alternate algorithm, called RMP2, that uses modern automatic differentiation tools (such as TensorFlow and PyTorch) to compute RMPflow policies. Our new design retains the strengths of RMPflow while bringing in advantages from automatic differentiation, including 1) easy programming interfaces to designing complex transformations; 2) support of general directed acyclic graph (DAG) transformation structures; 3) end-to-end differentiability for policy learning; 4) improved computational efficiency. Because of these features, RMP2 can be treated as a structured policy class for efficient robot learning which is suitable encoding domain knowledge. Our experiments show that using structured policy class given by RMP2 can improve policy performance and safety in reinforcement learning tasks for goal reaching in cluttered space.
BibTeX
@inproceedings{li2021rmp,
  author = {Anqi Li and Ching-An Cheng and Muhammad Asif Rana and Man Xie and Karl Van Wyk and Nathan D. Ratliff and Byron Boots},
  title = {{RMP2: A Structured Composable Policy Class for Robot Learning}},
  booktitle = {Robotics: Science and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=C5Y7AlUAtk},
}
Figure