H-MaP: An Iterative and Hybrid Sequential Manipulation Planner

Berk Cicek, Cankut Bora Tuncer, Busenaz Kerimgil, Ozgur S. OguzIEEE RA-L 2024 · OpenReview

TL;DR

This paper introduces H-MaP, a novel hybrid planner that tackles complex robotic manipulation tasks involving sequential actions and dynamic contact mode switches. By decoupling object trajectory planning from robot motion planning and employing a learning-informed contact sampling strategy, H-MaP significantly improves robustness and scalability, especially in challenging, constrained environments.

What Problem Does This Paper Address?

Robots performing complex manipulation tasks, like using tools or coordinating multiple arms, face significant challenges. Traditional methods often struggle with:

  • Sequential Actions: Chaining together distinct steps (e.g., pick up a tool, then use it to push an object) requires a planner that can handle diverse task configurations.
  • Dynamic Contact Mode Switches: Many tasks involve changing how the robot interacts with an object (e.g., from sliding to grasping). Explicitly planning for these transitions is crucial but difficult.
  • High Dimensionality: Planning for both the robot's configuration and the object's state simultaneously leads to enormous search spaces, hindering efficiency and scalability.
  • Local Minima: Optimization-based planners can get stuck in suboptimal solutions, especially in constrained environments with complex geometries.
  • Tool Use and Auxiliary Objects: Handling tasks that involve manipulating objects with tools or using intermediate objects is often not well-addressed by existing methods.

H-MaP aims to overcome these limitations by providing a unified approach to sequential and constrained manipulation planning.

Methods

H-MaP employs a three-phase iterative architecture:

  1. Waypoint Generation (Object-Based RRT): Instead of planning in the full robot-object configuration space, H-MaP first plans a trajectory for the target object in 3D space, ignoring the robot's kinematics. This significantly reduces dimensionality. A Bi-RRT algorithm is used for this, treating the object as a free-moving body. Waypoints are then extracted from this object trajectory.
  2. Contact Point Generation (Hybrid Learning & Sampling): For each object waypoint, H-MaP determines feasible physical interaction points between the robot (or tool) and the object. This phase uses a novel hybrid approach:
  3. Learning-Informed: An Artificial Neural Network (ANN), trained on successful manipulation examples, proposes initial contact points based on the current scene configuration.
  4. Sampling-Based Validation: If the ANN-proposed contact point is not feasible (checked via inverse kinematics), the system falls back to uniform sampling over the object's point cloud. This combination leverages learned priors for efficiency while retaining robustness through sampling.
  5. Optimization-Based Motion Planning (KOMO): Finally, a motion planner (KOMO, a non-linear programming solver) generates the robot's trajectory between consecutive waypoints, incorporating the determined contact points and kinematic modes. This phase ensures kinematically feasible and stable motion.

The system is iterative: if a path to a waypoint is infeasible, it can trigger replanning or fall back to waypoint generation. It also handles dynamic obstacles by recursively replanning.

Datasets

  • Training Data: A dataset of successful manipulation examples was curated from task-specific simulation environments to train the ANN for informed contact point proposals.
  • Evaluation Data: Seven diverse manipulation tasks were used for evaluation, including object-through-tunnel, tool-assisted retrieval, latch manipulation, obstacle navigation, and bookshelf organization. These tasks were simulated using the RAI simulator and validated on real robots.

Key Insights and Ideas

  • Decoupling Object and Robot Planning: Planning the object's path independently first is a powerful way to reduce the dimensionality of the problem, making it more tractable.
  • Hybrid Contact Point Generation: Combining learned priors (ANN) with robust sampling is key to efficiently finding feasible contact points, especially in complex scenarios. This balances speed and reliability.
  • Iterative Refinement: The multi-phase, iterative nature of H-MaP allows it to recover from initial planning failures and adapt to complex constraints.
  • Beyond Predefined Primitives: The approach demonstrates that explicit, high-level action definitions are not always necessary for complex manipulation; low-level motion planners can implicitly handle these by focusing on waypoints and contact points.

Key Results

  • Superior Performance: H-MaP achieved a 100% success rate across all seven diverse tasks over 10 random trials, outperforming baseline methods like KOMO, Bi-RRT, CMGMP, and LGP.
  • Robustness in Constrained Environments: H-MaP successfully handled tasks where traditional optimization methods failed due to local minima or scalability issues.
  • Efficiency Gains: The learning-informed contact sampling reduced planning time significantly, with improvements up to 30% for complex tasks like the bookshelf scenario.
  • Real-World Validation: The planner's effectiveness was demonstrated through successful execution on a Franka Panda robot for tasks like bookshelf manipulation and obstacle removal.

Limitations and Future Work

  • Action Limitations: In certain scenarios (e.g., flip-card), the decoupled planning might lead to premature pick actions, resulting in infeasible motion. Future work could involve informing waypoint generation with configuration-based action constraints.
  • Waypoint Quality Assumption: The solver assumes RRT-generated waypoints are viable. Poor waypoint generation could lead to solver failure. While mitigated by waypoint regeneration, this remains a potential issue.
  • Kinematic Focus: The current implementation is limited to kinematic systems. Future work aims to incorporate system dynamics and implement controllers for real-world execution.
  • Backtracking Strategies: The paper suggests future work on more advanced backtracking strategies for even greater robustness.

Summary updated

Abstract

This study introduces the Hybrid Sequential Manipulation Planner (H-MaP), a novel approach that iteratively does motion planning using contact points and waypoints for complex sequential manipulation tasks in robotics. Combining optimization-based methods for generalizability and sampling-based methods for robustness, H-MaP enhances manipulation planning through active contact mode switches and enables interactions with auxiliary objects and tools. This framework, validated by a series of diverse physical manipulation tasks and real-robot experiments, offers a scalable and adaptable solution for complex real-world applications in robotic manipulation.
BibTeX
@article{cicek2024h,
  author = {Berk Cicek and Cankut Bora Tuncer and Busenaz Kerimgil and Ozgur S. Oguz},
  title = {{H-MaP: An Iterative and Hybrid Sequential Manipulation Planner}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=47YLKie3Lt},
}
Figure