OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation

Yifan You, Lin Shao, Toki Migimatsu, Jeannette Bohg

TL;DR

OmniHang is a general-purpose framework that enables robots to hang a wide variety of everyday objects—from mugs and spatulas to bags and tools—onto diverse supports like hooks and racks. By combining contact-point correspondence prediction with a neural collision estimator, the system can determine stable hanging poses and plan collision-free paths using only the partial, noisy point clouds provided by standard depth cameras.

What Problem Does This Paper Address?

Hanging an object is a deceptively complex manipulation task. Unlike simple pick-and-place, hanging requires a deep understanding of topology (e.g., finding a hole or a loop) and support relationships (how gravity and friction will keep the object in place).

Most existing robotic solutions are "over-specialized"—they might work perfectly for hanging a specific mug on a specific rack but fail if the mug's handle is shaped differently. Furthermore, traditional motion planners require full 3D models of objects to avoid collisions. In the real world, robots only see "partial point clouds" (the side of the object facing the camera), making it difficult to know if the back of a spatula is hitting the wall or if a hook is properly threaded through a handle.

Methods

The authors propose a modular pipeline that breaks the problem into "Where to hang" and "How to hang."

1. Goal Pose Prediction (The "Where")

This happens in three distinct stages:

  • Stage One (Rough Pose): Two PointNet++ encoders process the partial point clouds of the object and the support. To account for the fact that an object might be hung in multiple ways, the model incorporates a random noise vector (similar to a GAN) to explore different possible stable poses.
  • Stage Two (Contact Point Matching): This is the core innovation. Instead of looking for "semantic" parts (like a handle), the model predicts which specific points on the object will touch the support. By aligning these predicted contact points, the system significantly refines the object's translation.
  • Stage Three (Local Refinement): The system uses Reinforcement Learning (RL) and the Cross-Entropy Method (CEM) to perform "micro-adjustments." This stage polishes the pose to ensure the object won't slip or fall once released.

2. Neural Path Planning (The "How")

To move the object from the gripper to the hook without crashing, the authors developed a Neural Collision Estimator (NCE). While standard planners (like RRT-Connect) usually need perfect geometric data, the NCE is a deep network trained to predict collisions directly from partial point clouds. It essentially "fills in the blanks" of what the camera cannot see to ensure a safe path.

Datasets

The researchers introduced a massive new synthetic benchmark:

  • The Hanging Pose Dataset: Contains 340 unique objects and 72 supporting items (hooks, racks, pegboards).
  • Scale: 19,299 pairs of object/support combinations, annotated with stable poses and ground-truth contact points derived from the PyBullet physics engine.
  • Diversity: Includes categories like bags, caps, hangers, utensils, headphones, knives, mugs, racquets, scissors, and wrenches.

Key Insights and Ideas

  • Contact Points > Semantic Keypoints: Previous work tried to identify "the center of the handle." OmniHang argues that predicting contact correspondences is more robust. A hole in a wrench and the strap of a bag are geometrically different but functionally identical when it comes to hanging; contact points capture this functional similarity.
  • The "Warm Start" Advantage: Training an RL agent to hang objects from scratch is incredibly slow. By using Stages 1 and 2 to provide a "good guess," the authors sped up the RL refinement (Stage 3) by a factor of six.
  • Learning to "See" Collisions: The Neural Collision Estimator proves that robots don't necessarily need a perfect 3D mesh to plan paths. By training on thousands of simulated examples, the network learns the "intuition" of which relative poses are likely to result in a collision.

Key Results

  • High Success Rate: The full three-stage pipeline achieved a 68.3% success rate in predicting stable poses across all object categories.
  • Ablation Importance: Without the Stage 2 contact matching, success dropped significantly. For example, Stage 1 alone only managed a 36% success rate, highlighting that "rough" estimation isn't enough for the precision required in hanging.
  • Planning Superiority: The Neural Collision Estimator (NCE) achieved an F1 score of 52.1% for finding feasible paths, significantly outperforming a heuristic baseline (38.7%) that struggled with partial observations.
  • Generalization: The system successfully hung objects it had never seen during training, demonstrating true class-agnostic manipulation.

Limitations and Future Work

  • Simulation vs. Reality: The current results are primarily in simulation (PyBullet). While the use of partial point clouds prepares the system for real-world sensors, physical robot experiments are the necessary next step.
  • Rigid Bodies Only: The current model assumes objects are rigid. It does not yet account for deformable objects like soft bags or clothing, which change shape when hung.
  • Dynamic Hanging: The system plans a path to a static pose. Future work could explore "dynamic" hanging, such as tossing a ring onto a hook or sliding an object along a rail.

Summary updated

Abstract

In this paper, we explore whether a robot can learn to hang arbitrary objects onto a diverse set of supporting items such as racks or hooks. Endowing robots with such an ability has applications in many domains such as domestic services, logistics, or manufacturing. Yet, it is a challenging manipulation task due to the large diversity of geometry and topology of everyday objects. In this paper, we propose a system that takes partial point clouds of an object and a supporting item as input and learns to decide where and how to hang the object stably. Our system learns to estimate the contact point correspondences between the object and supporting item to get an estimated stable pose. We then run a deep reinforcement learning algorithm to refine the predicted stable pose. Then, the robot needs to find a collision-free path to move the object from its initial pose to stable hanging pose. To this end, we train a neural network based collision estimator that takes as input partial point clouds of the object and supporting item. We generate a new and challenging, large-scale, synthetic dataset annotated with stable poses of objects hung on various supporting items and their contact point correspondences. In this dataset, we show that our system is able to achieve a 68.3% success rate of predicting stable object poses and has a 52.1% F1 score in terms of finding feasible paths. Supplemental material and videos are available on our project webpage.
BibTeX
@inproceedings{you2021omnihang,
  author = {Yifan You and Lin Shao and Toki Migimatsu and Jeannette Bohg},
  title = {{OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9560971},
}

Loading related content...

Figure