Abstract
Robot learning often faces a fundamental trade-off between the safety and stability of analytical control and the flexibility of machine learning. We present RMP$^2$, a practical implementation of the RMPflow framework designed for modern robot learning pipelines. By reformulating RMPflow's message-passing as a closed-form solution powered by automatic differentiation (Autodiff), RMP$^2$ allows for the seamless integration of control-theoretic priors into end-to-end reinforcement learning. Our results demonstrate that RMP$^2$ enables zero-collision exploration and significantly accelerates convergence in complex manipulation tasks.
Introduction
In robotics, researchers often choose between analytical control, which is safe but difficult to design for complex tasks, and machine learning, which is flexible but often unsafe and data-hungry. RMPflow was introduced to bridge this gap by combining multiple "Riemannian Motion Policies" (RMPs)—such as goal-reaching, collision avoidance, and joint-limit centering—into a single acceleration-based policy.
However, the original RMPflow implementation posed significant challenges for learning. It required manual definition of rigid tree structures and used a custom message-passing algorithm that was difficult to differentiate through. RMP$^2$ addresses these issues by re-imagining the framework through the lens of Automatic Differentiation, making it compatible with standard deep learning tools like PyTorch and TensorFlow.
The RMP$^2$ Framework
The key technical insight of RMP$^2$ is that the RMPflow message-passing algorithm is mathematically equivalent to solving a specific weighted least-squares optimization problem. Instead of manual "pushforward" and "pullback" steps, RMP$^2$ derives a closed-form solution for the robot's acceleration.
To compute this efficiently, the authors leverage three standard Autodiff primitives:
gradient: For scalar outputs.jacobian: To compute task-map derivatives.jvp(Jacobian-vector product): To compute velocities and curvature terms without explicitly forming massive matrices.
By using double backpropagation, RMP$^2$ achieves the same $O(N)$ linear time complexity as the original RMPflow but with a simpler user interface and support for Directed Acyclic Graphs (DAGs), which is essential for complex robots with shared kinematic chains.
Experimental Setup
The researchers evaluated RMP$^2$ using two simulated environments in PyBullet:
- Three-Link Planar Robot: A 2D arm tasked with reaching goals while avoiding cylindrical obstacles across three difficulty levels (varying goal range and obstacle density).
- Franka Emika Panda: A 7-degree-of-freedom (DOF) industrial manipulator in a 3D workspace, tasked with reaching goals while avoiding spherical obstacles.
Key Results and Discussion
Safety as a Prior
A standout feature of RMP$^2$ is the use of Residual RMP learning. Instead of training a neural network from scratch, the learner optimizes a "residual" policy on top of a hand-designed RMP prior. Because the structured policy class has built-in collision avoidance, the robot maintains safety even during the earliest, most random phases of reinforcement learning.
In the Franka robot task, the RMP-Residual policy maintained zero collisions throughout the entire training process. In contrast, standard neural network policies suffered from frequent crashes during exploration.
Convergence and Efficiency
RMP$^2$ significantly accelerates learning. In the most challenging 3-link arm environment, the RMP-Residual approach improved performance by more than 100% compared to standard neural networks within the same training budget. Furthermore, benchmarks showed that RMP$^2$ is significantly faster than naïve Autodiff implementations, especially as the complexity of the robot's task map increases.
Limitations and Future Work
While RMP$^2$ simplifies implementation, it currently focuses on acceleration-based (kinematic) control. Extending this to full torque-level dynamics for highly dynamic systems like quadrupeds remains an open challenge. Additionally, while the framework is safer, high-dimensional tasks like the 7-DOF Franka experiments suggest that structured policies may require further tuning of learning hyperparameters to reach optimal rewards within short training windows.
Conclusion
RMP$^2$ provides a practical, high-performance bridge between control theory and modern robot learning. By embedding safety and stability guarantees directly into the policy class, it enables more robust and efficient training for complex robotic tasks.
Summary updated