Complex In-Hand Manipulation Via Compliance-Enabled Finger Gaiting and Multi-Modal Planning

Andrew S. Morgan, Kaiyu Hang, Bowen Wen, Kostas E. Bekris, Aaron M. Dollar

TL;DR

This paper presents a novel approach to in-hand manipulation that allows robots to reorient objects in 3D space (SO(3)) using "finger gaiting" – the dynamic repositioning of contacts. By leveraging the inherent compliance of an underactuated robotic hand and a vision-based object tracker, the system can autonomously achieve complex object reorientations without a support surface, even in the presence of disturbances.

What Problem Does This Paper Address?

Traditional in-hand manipulation methods often rely on fixed or rolling contacts, which severely limit the object's reachable workspace. Finger gaiting, the act of breaking and remaking contacts, offers a way to overcome these limitations and access a much larger workspace. However, implementing finger gaiting has been computationally challenging, requiring precise control of forces and stability during contact transitions. This paper tackles the difficulty of realizing practical, autonomous finger gaiting for full 3D object reorientation, especially against gravity and without relying on external support.

Methods

The core of the proposed method is a multi-modal planning approach that leverages the compliance of the robotic hand. Instead of explicitly modeling complex contact dynamics, the system exploits the hand's passive adaptive properties to simplify planning.

  • Compliance-Enabled Finger Gaiting: The system uses an underactuated, compliant hand with four fingers. Compliance allows the hand to absorb small errors and "inflate" the regions where contacts can be safely switched, making transitions smoother and less prone to failure.
  • Multi-Modal Planning: The planner operates by defining "modes" of manipulation, each corresponding to a specific set of constraints (e.g., a particular contact configuration). The planner searches for paths that transition between these modes to achieve the desired object orientation.
  • SO(3) Planning: For orientation control, the planner focuses on rotations around two orthogonal axes (z and x) to achieve any arbitrary 3D orientation. It uses a bi-directional search: a backward pass to expand the goal manifold and a forward pass to find a path from the start orientation to this manifold.
  • Online Replanning and Recovery: A low-latency 6D pose object tracker provides real-time feedback. If the object deviates from its planned trajectory or enters an unstable state (e.g., too far from the workspace center), the planner updates its strategy online to recover the object's pose. This includes a dedicated recovery phase for translational adjustments.
  • Vision-Based Control: The system relies on visual feedback from an RGBD camera and a robust 6D pose object tracker, eliminating the need for tactile sensors or joint encoders on the hand.

Datasets

Not discussed in the paper. The paper focuses on experimental validation using real-world objects and a robotic system.

Key Insights and Ideas

  • Compliance as a Planning Simplifier: The paper demonstrates that by embracing the passive compliance of an underactuated hand, complex manipulation problems like finger gaiting can be reframed from force/stability control to a motion-only control paradigm. This significantly reduces the computational burden.
  • "Inflated" Contact Switching Regions: Compliance effectively creates "safe zones" for breaking and remaking contacts, making the transitions more robust and forgiving of minor errors. This is quantified by a parameter 'p' representing the inflation threshold.
  • Unified Motion-Only Control: The system unifies orientation and translation control by treating them as distinct "modes" that can be transitioned between. This allows for a more systematic approach to complex manipulation tasks.
  • Autonomous SO(3) Control without Support: Achieving full 3D orientation control of an object within a hand, against gravity, and without any external support is a significant step towards more generalizable robotic manipulation.

Key Results

  • Successful SO(3) Reorientation: The system successfully demonstrated complex in-hand reorientation of both convex (cube, sphere, toy truck) and non-convex (Stanford Bunny, toy duck) objects.
  • Robustness to Perturbations: The system showed resilience to unmodeled perturbations, replanning trajectories online to recover from disturbances and achieve the goal orientation within a small error margin (e.g., within 6° for a 205-second trajectory).
  • Repeatability: Trajectories for controlled dimensions (roll, yaw) were highly repeatable across multiple trials, while uncontrolled dimensions (translation, pitch) showed more drift, as expected.
  • Efficiency: While not explicitly optimized for speed, the system achieved object reorientation tasks within reasonable timeframes (e.g., ~110 seconds for a toy truck with multiple recovery phases).
  • First Autonomous Full SO(3) Control: The authors claim this is the first work to autonomously achieve full SO(3) control of objects within-hand via finger gaiting without a support surface.

Limitations and Future Work

  • Hand Design Dependency: The current method is tied to the specific capabilities of the Yale OpenHand Model Q, particularly its limited translational capabilities (no x-axis translation). The authors suggest that a hand with y-axis rotation capabilities would shorten trajectories.
  • "Inflation" Parameter (p): The parameter 'p', which defines the safe contact switching region, is estimated through experimentation and requires further theoretical investigation.
  • Geometry-Focused Modes: Developing modal actions and transitions that are more specifically tailored to object geometry could improve performance.
  • Object-Agnostic Tracking: While the controller is object-agnostic, the current tracking schema requires an object CAD model for training. Future work aims to combine object-agnostic tracking methods with the controller for instant application to novel objects.
  • Speed: The manipulation actions were noted to be slower than some other related works, suggesting room for optimization.

Summary updated

Abstract

Constraining contacts to remain fixed on an object during manipulation limits the potential workspace size, as motion is subject to the hand’s kinematic topology. Finger gaiting is one way to alleviate such restraints. It allows contacts to be freely broken and remade so as to operate on different manipulation manifolds. This capability, however, has traditionally been difficult or impossible to practically realize. A finger gaiting system must simultaneously plan for and control forces on the object while maintaining stability during contact switching. This letter alleviates the traditional requirement by taking advantage of system compliance, allowing the hand to more easily switch contacts while maintaining a stable grasp. Our method achieves complete <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$SO(3)$</tex-math></inline-formula> finger gaiting control of grasped objects against gravity by developing a manipulation planner that operates via orthogonal safe modes of a compliant, underactuated hand absent of tactile sensors or joint encoders. During manipulation, a low-latency 6D pose object tracker provides feedback via vision, allowing the planner to update its plan online so as to adaptively recover from trajectory deviations. The efficacy of this method is showcased by manipulating both convex and non-convex objects on a real robot. Its robustness is evaluated via perturbation rejection and long trajectory goals. To the best of the authors’ knowledge, this is the first work that has autonomously achieved full <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$SO(3)$</tex-math></inline-formula> control of objects within-hand via finger gaiting and without a support surface, elucidating a valuable step towards realizing true robot in-hand manipulation capabilities.
BibTeX
@article{morgan2022complex,
  author = {Andrew S. Morgan and Kaiyu Hang and Bowen Wen and Kostas E. Bekris and Aaron M. Dollar},
  title = {{Complex In-Hand Manipulation Via Compliance-Enabled Finger Gaiting and Multi-Modal Planning}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  url = {https://openreview.net/forum?id=hzQx12hn03},
}
Figure