Towards Coordinated Robot Motions: End-to-End Learning of Motion Policies on Transform Trees

Muhammad Asif Rana, Anqi Li, Dieter Fox, Sonia Chernova, Byron Boots, Nathan D. Ratliff

TL;DR

This paper introduces a novel end-to-end learning framework for generating coordinated robot motions that simultaneously satisfy multiple tasks. By leveraging human demonstrations and a structured policy inspired by RMPflow, the method learns to combine task-specific policies in a way that guarantees stable and effective motion, outperforming existing independent learning approaches.

What Problem Does This Paper Address?

Generating robot motion that can accomplish multiple objectives at once is a significant challenge. Robots often need to perform tasks like reaching a target location while simultaneously avoiding obstacles or maintaining a specific posture. These tasks are frequently best described in different "spaces" (e.g., end-effector position in workspace, joint angles in configuration space). The core difficulty lies in coordinating these diverse task specifications, which are inherently linked by the robot's physical constraints. Existing methods often struggle to effectively combine these tasks, leading to suboptimal or unstable motions.

Methods

The authors propose a velocity-control motion generation algorithm that builds upon the RMPflow framework. Instead of learning separate policies for each task and then combining them, they introduce an end-to-end learning objective. This objective directly penalizes deviations in the task spaces, effectively learning how to balance and coordinate multiple subtask policies simultaneously.

Key methodological contributions include:

  • Transform Trees: A hierarchical structure to represent the relationships between the robot's configuration space and various subtask spaces. This allows for efficient computation and organization of policies.
  • End-to-End Learning Objective: A novel objective function that is differentiable through the motion generation process. This enables learning to account for the geometric constraints between tasks during the training phase.
  • Stable Subtask Policies: The learned subtask policies are parameterized using a chain of diffeomorphisms and latent importance weight matrices. This design choice ensures that the resulting combined motion is Lyapunov stable, a critical property for safe robot operation.
  • Flexibility in Learning: The framework allows for a mix of learned and hand-designed subtask policies, giving users control over which behaviors are learned from data and which are pre-defined (e.g., for collision avoidance or joint damping).

Datasets

The paper evaluates its approach on three manipulation tasks performed by a 7-DOF Rethink Sawyer robot: inspection, placing-1, and placing-2. For each task, human subjects provided multiple demonstrations through kinesthetic teaching. The authors do not introduce new benchmark datasets but rather use these specific task setups to demonstrate their method's effectiveness.

Key Insights and Ideas

  • Coordinated Learning is Crucial: Learning multiple subtask policies simultaneously, rather than independently, leads to significantly better performance. This is because the learning process can account for how policies interact and influence each other due to robot kinematics.
  • Task-Space Deviation is Key: Focusing the learning objective on minimizing errors in the task spaces (where the tasks are naturally defined) is more effective than simply trying to match demonstrated joint-space trajectories. Humans demonstrate based on task goals, not necessarily optimal joint configurations.
  • Stability Guarantees Matter: The proposed method's ability to guarantee Lyapunov stability for the generated motions is a significant advantage, especially for real-world robotic applications where safety is paramount.
  • Diffeomorphisms for Expressiveness: Using chains of diffeomorphisms to parameterize subtask policies allows for learning complex, non-linear mappings from latent spaces to task spaces, enabling the representation of intricate motions.

Key Results

The proposed coordinated learning approach significantly outperforms baseline methods (independent learning and single-link learning) across the evaluated tasks.

  • Quantitative Improvements: The method achieves lower mean position and orientation errors compared to baselines on the "placing-1" and "placing-2" tasks.
  • Generalization Success: The approach demonstrates a higher generalization success rate over 10 novel initial configurations, indicating its robustness and ability to adapt to unseen starting points.
  • Qualitative Validation: Visualizations show that the learned policies successfully reproduce demonstrated motions, including complex trajectories involving object manipulation and precise end-effector poses.

Limitations and Future Work

  • Limitations:
    • The paper notes that while the independent learning version of their approach might perform worse than a single-link learning variant, this is attributed to potential conflicts between independently learned subtask policies.
  • For the "inspection" task, the quantitative error differences between the coordinated approach and baselines were smaller, suggesting that kinematic constraints might implicitly enforce some coordination. However, for highly constrained tasks, even small errors can lead to failures, highlighting the importance of the coordinated approach.
  • Future Work:
    • The authors suggest that their framework is not limited to learning robot poses and can be extended to other types of control points or even partial poses.
  • The flexibility of the framework allows for the inclusion of hand-designed policies, which could be further explored for specific safety-critical behaviors.
  • The paper implicitly points towards exploring more complex task decompositions and robot configurations.

Summary updated

Abstract

Generating robot motion that fulfills multiple tasks simultaneously is challenging due to the geometric constraints imposed on the robot. In this paper, we propose to solve multi-task problems through learning structured policies from human demonstrations. Our structured policy is inspired by RMPflow, a framework for combining subtask policies on different spaces. The policy structure provides the user an interface to 1) specifying the spaces that are directly relevant to the completion of the tasks, and 2) designing policies for certain tasks that do not need to be learned. We derive an end-to-end learning objective that is suitable for the multi-task problem, emphasizing the distance between generated motions and demonstrations measured on task spaces. Furthermore, the motion generated from the learned policy class is guaranteed to be stable. We validate the effectiveness of our proposed learning framework through qualitative and quantitative evaluations on three robotic tasks on a 7-DOF Rethink Sawyer robot.
BibTeX
@inproceedings{rana2021towards,
  author = {Muhammad Asif Rana and Anqi Li and Dieter Fox and Sonia Chernova and Byron Boots and Nathan D. Ratliff},
  title = {{Towards Coordinated Robot Motions: End-to-End Learning of Motion Policies on Transform Trees}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=K0l2HRumhs},
}
Figure