Introduction
In modern robotics, a persistent tension exists between analytic control and machine learning. While control theory offers formal safety guarantees, it often struggles with the complexity of unstructured tasks. Conversely, Reinforcement Learning (RL) excels at learning complex behaviors but often operates as a "black box," requiring vast amounts of data and frequently resulting in unsafe behaviors during the exploration phase. This paper introduces RMP$^2$, a streamlined implementation of the RMPflow framework designed to bridge this gap by leveraging modern automatic differentiation tools.
Methodology: RMP2 and Automatic Differentiation
RMPflow is a multi-task control framework that decomposes complex robot motions into smaller "Riemannian Motion Policies" (RMPs). The original implementation relied on a manual message-passing algorithm on tree structures, which was computationally expensive and difficult to integrate with deep learning workflows.
RMP$^2$ re-imagines this framework by observing that the core "pullback" operation in RMPflow is mathematically equivalent to solving a weighted least-squares optimization problem. By utilizing standard automatic differentiation operators—such as Jacobian-vector products (JVP)—RMP$^2$ achieves several key advantages:
- Computational Efficiency: Memory footprint is reduced from $O(Nd^2)$ to $O(Nd)$, enabling linear scaling with task complexity.
- Structural Flexibility: It supports Directed Acyclic Graphs (DAGs) rather than just trees, allowing for more complex robot-sensor configurations.
- End-to-End Differentiability: The entire control stack can be treated as a differentiable layer within a neural network, facilitating "Residual RMP learning."
In Residual RMP learning, the system is initialized with hand-crafted safety policies (e.g., collision avoidance). A neural network then learns a "residual" on top of these priors to optimize task-specific performance without sacrificing the underlying safety constraints.
Experimental Evaluation
The authors evaluated RMP$^2$ using the PyBullet physics engine across two primary platforms: a 2D three-link robot arm and a 7-DOF Franka Emika Panda manipulator. These robots were tasked with reaching goals in cluttered environments containing multiple obstacles.
Results and Discussion
The experimental results highlight the power of using a structured policy class. As shown in the performance metrics, RMP-based policies significantly outperform standard "black-box" neural networks (NN) in both efficiency and safety.
Zero-Shot Safety and Performance
In the Franka robot experiments, RMP-based policies maintained near 100% safety throughout the entire training process. In contrast, standard neural networks exhibited high collision rates, particularly during early exploration. In the most challenging environments, the RMP-Residual policy achieved more than double the mean reward of a standard NN, demonstrating that safety priors do not necessarily come at the cost of performance.
As illustrated in Figure 3, even as the goal range increases and the task becomes more complex, the RMP-based methods (NN-RESIDUAL and RMP-RESIDUAL) maintain a high safety percentage, while the standard NN's safety fluctuates and remains significantly lower. This confirms that RMP$^2$ effectively encodes domain knowledge as a constraint that the learning process cannot easily violate.
Conclusion
RMP$^2$ transforms a complex control-theoretic tool into a flexible, differentiable policy class suitable for modern robot learning. By treating safety as a structural prior rather than a learned behavior, it enables robots to explore cluttered environments safely from day one. While it requires the definition of geometric task maps, the benefits in terms of training stability, safety, and final performance make it a compelling framework for real-world robotics applications.
Summary updated