OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation

Yifan You, Lin Shao, Toki Migimatsu, Jeannette Bohg

Abstract

In this paper, we explore whether a robot can learn to hang arbitrary objects onto a diverse set of supporting items such as racks or hooks. Endowing robots with such an ability has applications in many domains such as domestic services, logistics, or manufacturing. Yet, it is a challenging manipulation task due to the large diversity of geometry and topology of everyday objects. In this paper, we propose a system that takes partial point clouds of an object and a supporting item as input and learns to decide where and how to hang the object stably. Our system learns to estimate the contact point correspondences between the object and supporting item to get an estimated stable pose. We then run a deep reinforcement learning algorithm to refine the predicted stable pose. Then, the robot needs to find a collision-free path to move the object from its initial pose to stable hanging pose. To this end, we train a neural network based collision estimator that takes as input partial point clouds of the object and supporting item. We generate a new and challenging, large-scale, synthetic dataset annotated with stable poses of objects hung on various supporting items and their contact point correspondences. In this dataset, we show that our system is able to achieve a 68.3% success rate of predicting stable object poses and has a 52.1% F1 score in terms of finding feasible paths.

Introduction

Hanging an object is a deceptively complex manipulation task. Unlike simple pick-and-place, hanging requires a deep understanding of topology (e.g., finding a hole or a loop) and support relationships (how gravity and friction will keep the object in place). Most existing robotic solutions are over-specialized, failing when object geometry varies slightly. Furthermore, traditional motion planners require full 3D models, whereas real-world robots typically only have access to partial, noisy point clouds provided by standard depth cameras.

Methods

The OmniHang framework addresses these challenges with a modular pipeline that separates the problem into "where to hang" and "how to hang."

Goal Pose Prediction

The system determines the final stable pose through a three-stage process:

  1. Stage One (Rough Pose): PointNet++ encoders process partial point clouds of the object and support. A noise vector allows the model to explore multiple potential hanging configurations.
  2. Stage Two (Contact Point Matching): This core innovation predicts specific contact points on the object and support. Aligning these points refines the object's translation significantly.
  3. Stage Three (Local Refinement): Using Reinforcement Learning (RL) and the Cross-Entropy Method (CEM), the system performs micro-adjustments to ensure stability against gravity.
The three-stage pipeline for goal pose prediction, moving from rough estimation to contact point matching and local refinement.

Neural Path Planning

To reach the goal pose without collisions, OmniHang uses a Neural Collision Estimator (NCE). Unlike standard planners that require perfect geometry, the NCE is a deep network trained to predict collisions directly from partial point clouds, effectively "filling in the blanks" of occluded areas.

Experimental Evaluation

The researchers introduced a massive synthetic benchmark containing 340 unique objects and 72 supporting items, totaling nearly 20,000 annotated pairs across categories like bags, headphones, and tools.

Visualizing the three stages of pose prediction across different object geometries. Red and blue dots represent predicted contact points.

Key Insights

  • Contact Points > Semantic Keypoints: Predicting functional contact correspondences is more robust than identifying semantic parts like "handles," as it generalizes across diverse geometries (e.g., a bag strap vs. a wrench hole).
  • The "Warm Start" Advantage: Using Stages 1 and 2 to provide a "good guess" for the RL refinement in Stage 3 speeds up training by a factor of six.
  • Learning to "See" Collisions: The NCE demonstrates that robots can plan paths without perfect 3D meshes by learning the intuition of relative poses from simulated data.

Results and Conclusion

The full pipeline achieved a 68.3% success rate across all categories. Ablation studies showed that Stage 2 is critical; without contact matching, success rates for rough estimation alone dropped to 36%. The NCE achieved an F1 score of 52.1% for path planning, significantly outperforming heuristic baselines. While successful in simulation, future work remains to bridge the gap to real-world physical experiments and to handle non-rigid, deformable objects.

Summary updated

Abstract

In this paper, we explore whether a robot can learn to hang arbitrary objects onto a diverse set of supporting items such as racks or hooks. Endowing robots with such an ability has applications in many domains such as domestic services, logistics, or manufacturing. Yet, it is a challenging manipulation task due to the large diversity of geometry and topology of everyday objects. In this paper, we propose a system that takes partial point clouds of an object and a supporting item as input and learns to decide where and how to hang the object stably. Our system learns to estimate the contact point correspondences between the object and supporting item to get an estimated stable pose. We then run a deep reinforcement learning algorithm to refine the predicted stable pose. Then, the robot needs to find a collision-free path to move the object from its initial pose to stable hanging pose. To this end, we train a neural network based collision estimator that takes as input partial point clouds of the object and supporting item. We generate a new and challenging, large-scale, synthetic dataset annotated with stable poses of objects hung on various supporting items and their contact point correspondences. In this dataset, we show that our system is able to achieve a 68.3% success rate of predicting stable object poses and has a 52.1% F1 score in terms of finding feasible paths. Supplemental material and videos are available on our project webpage.
BibTeX
@inproceedings{you2021omnihang,
  author = {Yifan You and Lin Shao and Toki Migimatsu and Jeannette Bohg},
  title = {{OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9560971},
}
Figure