RMP2: A Structured Composable Policy Class for Robot Learning

Anqi Li, Ching-An Cheng, Muhammad Asif Rana, Man Xie, Karl Van Wyk, Nathan D. Ratliff, Byron Boots

Introduction

In modern robotics, a persistent tension exists between analytic control and machine learning. While control theory offers formal safety guarantees, it often struggles with the complexity of unstructured tasks. Conversely, Reinforcement Learning (RL) excels at learning complex behaviors but often operates as a "black box," requiring vast amounts of data and frequently resulting in unsafe behaviors during the exploration phase. This paper introduces RMP$^2$, a streamlined implementation of the RMPflow framework designed to bridge this gap by leveraging modern automatic differentiation tools.

Methodology: RMP2 and Automatic Differentiation

RMPflow is a multi-task control framework that decomposes complex robot motions into smaller "Riemannian Motion Policies" (RMPs). The original implementation relied on a manual message-passing algorithm on tree structures, which was computationally expensive and difficult to integrate with deep learning workflows.

RMP$^2$ re-imagines this framework by observing that the core "pullback" operation in RMPflow is mathematically equivalent to solving a weighted least-squares optimization problem. By utilizing standard automatic differentiation operators—such as Jacobian-vector products (JVP)—RMP$^2$ achieves several key advantages:

  1. Computational Efficiency: Memory footprint is reduced from $O(Nd^2)$ to $O(Nd)$, enabling linear scaling with task complexity.
  2. Structural Flexibility: It supports Directed Acyclic Graphs (DAGs) rather than just trees, allowing for more complex robot-sensor configurations.
  3. End-to-End Differentiability: The entire control stack can be treated as a differentiable layer within a neural network, facilitating "Residual RMP learning."

In Residual RMP learning, the system is initialized with hand-crafted safety policies (e.g., collision avoidance). A neural network then learns a "residual" on top of these priors to optimize task-specific performance without sacrificing the underlying safety constraints.

Experimental Evaluation

The authors evaluated RMP$^2$ using the PyBullet physics engine across two primary platforms: a 2D three-link robot arm and a 7-DOF Franka Emika Panda manipulator. These robots were tasked with reaching goals in cluttered environments containing multiple obstacles.

Results and Discussion

The experimental results highlight the power of using a structured policy class. As shown in the performance metrics, RMP-based policies significantly outperform standard "black-box" neural networks (NN) in both efficiency and safety.

Figure 2: Overview of simulation environments (3-link arm and Franka Panda) alongside mean reward and safety percentage metrics.

Zero-Shot Safety and Performance

In the Franka robot experiments, RMP-based policies maintained near 100% safety throughout the entire training process. In contrast, standard neural networks exhibited high collision rates, particularly during early exploration. In the most challenging environments, the RMP-Residual policy achieved more than double the mean reward of a standard NN, demonstrating that safety priors do not necessarily come at the cost of performance.

Figure 3: Safe percentage over time for the 'Large Goal Range' scenario, illustrating the consistent safety of RMP-based methods compared to the declining safety of standard NNs.

As illustrated in Figure 3, even as the goal range increases and the task becomes more complex, the RMP-based methods (NN-RESIDUAL and RMP-RESIDUAL) maintain a high safety percentage, while the standard NN's safety fluctuates and remains significantly lower. This confirms that RMP$^2$ effectively encodes domain knowledge as a constraint that the learning process cannot easily violate.

Conclusion

RMP$^2$ transforms a complex control-theoretic tool into a flexible, differentiable policy class suitable for modern robot learning. By treating safety as a structural prior rather than a learned behavior, it enables robots to explore cluttered environments safely from day one. While it requires the definition of geometric task maps, the benefits in terms of training stability, safety, and final performance make it a compelling framework for real-world robotics applications.

Summary updated

Abstract

We consider the problem of learning motion policies for acceleration-based robotics systems with a structured policy class specified by RMPflow. RMPflow is a multi-task control framework that has been successfully applied in many robotics problems. Using RMPflow as a structured policy class in learning has several benefits, such as sufficient expressiveness, the flexibility to inject different levels of prior knowledge as well as the ability to transfer policies between robots. However, implementing a system for end-to-end learning RMPflow policies faces several computational challenges. In this work, we re-examine the message passing algorithm of RMPflow and propose a more efficient alternate algorithm, called RMP2, that uses modern automatic differentiation tools (such as TensorFlow and PyTorch) to compute RMPflow policies. Our new design retains the strengths of RMPflow while bringing in advantages from automatic differentiation, including 1) easy programming interfaces to designing complex transformations; 2) support of general directed acyclic graph (DAG) transformation structures; 3) end-to-end differentiability for policy learning; 4) improved computational efficiency. Because of these features, RMP2 can be treated as a structured policy class for efficient robot learning which is suitable encoding domain knowledge. Our experiments show that using structured policy class given by RMP2 can improve policy performance and safety in reinforcement learning tasks for goal reaching in cluttered space.
BibTeX
@inproceedings{li2021rmp,
  author = {Anqi Li and Ching-An Cheng and Muhammad Asif Rana and Man Xie and Karl Van Wyk and Nathan D. Ratliff and Byron Boots},
  title = {{RMP2: A Structured Composable Policy Class for Robot Learning}},
  booktitle = {Robotics: Science and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=C5Y7AlUAtk},
}
Figure