Dexterous Manipulation Graphs

Silvia Cruciani, Christian Smith, Danica Kragic, Kaiyu Hang

TL;DR

The Dexterous Manipulation Graph (DMG) is a novel planning tool that allows robots with simple parallel grippers to perform complex in-hand object repositioning. By representing the object's surface as a graph of valid contact points and finger orientations, robots can plan a sequence of "pushes" from a second arm to move an object into a desired pose without needing high-fidelity physics simulations.

What Problem Does This Paper Address?

In robotics, "in-hand manipulation"—the ability to move an object once it’s already been picked up—is a notorious challenge. Humans do this effortlessly using many fingers (intrinsic dexterity), but most industrial and research robots use simple two-fingered parallel grippers.

To move an object inside such a gripper, you usually need either a very complex hand or a perfect mathematical model of friction and dynamics. Existing planning methods often struggle with complex object shapes or require "regrasping" (dropping and picking the object back up), which is slow and risky. The authors aim to bridge this gap by using a dual-arm system where one arm holds the object and the other acts as an external "finger" to nudge it into the right spot.

Methods

The core innovation is the Dexterous Manipulation Graph (DMG). The process follows these steps:

  1. Object Segmentation: The object’s 3D point cloud is divided into small areas called "supervoxels." The center of each area is a potential contact point.
  2. Graph Construction: An initial graph connects adjacent supervoxels. This is then refined by removing connections that would force a finger to move across a sharp edge (based on surface normals).
  3. Orientation Discretization: For every contact point, the system calculates which finger orientations are "admissible" (i.e., the finger won't collide with the object's body).
  4. The DMG: The final graph consists of nodes representing a (position, orientation) pair. An edge exists between nodes if the robot can move between them using one of two manipulation primitives:
  5. Translation: Sliding the contact point while keeping the object's orientation constant.
  6. Rotation: Keeping the contact point fixed while rotating the object.
  7. Planning & Execution: Using Dijkstra’s algorithm, the robot finds the shortest path in the DMG from the current grasp to the goal. This path is translated into velocities for a dual-arm robot (like the Baxter) using the Extended Cooperative Task Space (ECTS) framework.

Datasets

The paper does not use a standard large-scale dataset. Instead, it evaluates the DMG on several 3D mesh models of varying complexity, including:

  • A simple cube.
  • A "plug-shaped" object with non-convex features.
  • Various household objects used in the physical experiments with the Baxter robot.

Key Insights and Ideas

  • Extrinsic Dexterity via Dual Arms: Instead of building a more complex hand, the authors use the robot's second arm to provide the necessary forces to slide or rotate the object. This treats the second arm as a controllable "external support."
  • Geometry over Dynamics: By focusing on the geometric connectivity of the object's surface and collision-free orientations, the DMG avoids the need for the "high-fidelity dynamic models" (like precise friction coefficients) that often make other planners fail in the real world.
  • The Manipulability Matrix: The authors introduce a way to visualize how "manipulatable" an object is. By looking at the connected components of the DMG, they can identify which parts of an object allow for continuous repositioning and which would require a full regrasp.

Key Results

  • Physical Validation: The system was successfully tested on a Baxter robot. Despite Baxter's known "compliant" (springy) joints, the robot achieved repositioning accuracy within 10mm and 5 degrees of the target.
  • Path Efficiency: The DMG successfully planned paths for non-convex objects where a straight-line path would have caused a collision between the gripper and the object.
  • Coordination Modes: They found that keeping the "holding" arm fixed while the "pushing" arm moved (asymmetrical coordination) was more precise than moving both arms simultaneously, due to the mechanical play in the robot's joints.

Limitations and Future Work

  • No Roto-translation: Currently, the DMG treats rotation and translation as separate steps. Combining them into a single fluid motion (roto-translation) is computationally harder but would lead to faster manipulation.
  • Regrasping Integration: The current DMG only works for motions that don't require releasing the object. The authors plan to integrate "regrasping nodes" to allow the robot to transition between different connected components of the graph.
  • Multi-fingered Hands: While designed for parallel grippers, the authors suggest the DMG framework could be extended to robots with three or more fingers by coordinating multiple paths on the graph simultaneously.
  • Sensing: The current implementation relies on AprilTags for object tracking; future versions could benefit from more robust tactile or visual feedback.

Summary updated

Abstract

We propose the Dexterous Manipulation Graph as a tool to address in-hand manipulation and reposition an object inside a robot's end-effector. This graph is used to plan a sequence of manipulation primitives so to bring the object to the desired end pose. This sequence of primitives is translated into motions of the robot to move the object held by the end-effector. We use a dual arm robot with parallel grippers to test our method on a real system and show successful planning and execution of in-hand manipulation.
BibTeX
@inproceedings{cruciani2018dexterous,
  author = {Silvia Cruciani and Christian Smith and Danica Kragic and Kaiyu Hang},
  title = {{Dexterous Manipulation Graphs}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  doi = {10.1109/iros.2018.8594303},
}

Loading related content...

Figure