TL;DR
This paper introduces a novel depth-based framework for robustly estimating the 6D pose of objects held within an adaptive robot hand, even under severe occlusion. By first estimating the hand's configuration and then using this to prune occluding hand parts, the method achieves high accuracy and computational efficiency, making it suitable for challenging in-hand manipulation tasks.
What Problem Does This Paper Address?
Accurately determining an object's 6D pose (position and orientation) is crucial for many robotic manipulation tasks, especially those involving grasping and in-hand manipulation. However, when a robot's hand, particularly an adaptive one, grasps an object, it often occludes a significant portion of it. This occlusion makes traditional RGB-based methods struggle, especially with texture-less objects or when the hand and object have similar appearances. Existing methods often rely on pre-grasp poses, which are not always indicative of the post-grasp pose due to contact effects. Furthermore, adaptive hands, with their complex and often un-sensored finger configurations, add another layer of difficulty. This paper tackles the challenge of estimating an object's pose after it has been grasped by an adaptive hand, in the presence of severe occlusion, using only depth data.
Methods
The proposed framework comprises three key components:
- Hand State Estimation: This is a crucial first step. The system uses a Particle Swarm Optimization (PSO) approach to estimate the configuration of the adaptive hand (specifically, the Yale Hand T42). PSO is chosen for its robustness to local optima and its ability to handle non-differentiable cost functions. The cost function penalizes collisions and aims to maximize the Largest Common Pointset (LCP) score between the hand model and the observed point cloud. This estimated hand configuration is then used to segment the point cloud, effectively removing the hand's points and leaving only the object.
- Object Pose Hypotheses Generation: With the hand points removed, the system generates a set of potential 6D poses for the object. This is achieved through a robust global pointset registration method. It samples 4-point co-planar bases from the object's point cloud and searches for congruent sets on the object's 3D model. Heuristics guided by the hand state are used to improve sampling efficiency. These matches are then used to generate initial pose hypotheses.
- Pose Hypothesis Pruning and Selection: To refine the hypotheses, the framework employs physical reasoning. It prunes poses that would result in collisions between the object and the hand, or that are physically implausible (e.g., the object floating away from the hand). Furthermore, it leverages scene-level physics reasoning by comparing rendered depth images with the observed depth data to identify the most likely pose. This process is computationally efficient due to the pre-computed hand SDF.
Datasets
The authors introduce a new benchmark dataset for in-hand 6D object pose estimation. This dataset includes:
- Synthetic Data: Generated using a simulator, featuring 6 different objects and 2 adaptive hands (painted blue and white) in various poses. This dataset provides ground-truth object pose and semantic segmentation.
- Real-World Data: Collected using an Intel RealSense SR300 RGB-D camera, featuring 4 types of objects held by 2 Yale T42 adaptive hands. This dataset is challenging due to severe occlusions and noise inherent in consumer-grade depth sensors.
The paper notes that existing datasets are primarily for objects in human hands, making their new dataset valuable for robot manipulation research.
Key Insights and Ideas
- Hand State Estimation is Key to Occlusion Handling: The most significant insight is that accurately estimating the robot hand's configuration is paramount for effectively dealing with severe occlusions. By knowing where the hand is and how it's configured, the system can intelligently prune the occluding parts of the point cloud, revealing the object for subsequent pose estimation.
- PSO for Non-Differentiable Hand Pose: Using Particle Swarm Optimization for hand state estimation is a smart choice, as it bypasses the need for differentiable hand models or cost functions, which can be difficult to obtain for adaptive hands.
- Leveraging Physical Constraints for Pruning: Incorporating physical reasoning (collision checking, hand-object interaction) significantly reduces the number of false positive pose hypotheses, leading to more robust results.
- Depth-Based Robustness: The reliance on depth data makes the approach less susceptible to lighting variations and textureless objects, which are common challenges for RGB-based methods.
Key Results
- High Accuracy: The proposed method achieves high accuracy, with a recall of 95.33% on the simulation dataset and 83.52% on the real dataset (with an error threshold of 5mm).
- Computational Efficiency: The framework is computationally efficient, returning the object pose and hand state in approximately 0.5 to 0.7 seconds, making it suitable for real-time applications or integration with tracking systems.
- Superior Performance: The method outperforms existing state-of-the-art methods, including Super4PCS and learning-based approaches like DOPE and AAE, particularly in challenging, highly-occluded scenarios.
- Ablation Study Validation: An ablation study demonstrates the critical contribution of each component (PSO hand pose estimation, heuristic sampling, hypothesis pruning) to the overall performance.
Limitations and Future Work
- Dependency on Hand Model: The method relies on having an accurate CAD model of the adaptive hand.
- Consumer-Depth Sensor Noise: While robust, performance can still be affected by noise in consumer-grade depth sensors, as seen in some failure cases.
- Future Work: The authors suggest integrating their method with tracking-based approaches to provide initialization or recovery from lost tracking. They also highlight the potential usefulness of the estimated hand state for control purposes.
Summary updated