Abstract
We present a new dataset for 6-DoF pose estimation of known objects, with a focus on robotic manipulation research. We propose a set of toy grocery objects, whose physical instantiations are readily available for purchase and are appropriately sized for robotic grasping and manipulation. We provide 3D scanned textured models of these objects, suitable for generating synthetic training data, as well as RGBD images of the objects in challenging, cluttered scenes exhibiting partial occlusion, extreme lighting variations, multiple instances per image, and a large variety of poses. Using semi-automated RGBD-to-model texture correspondences, the images are annotated with ground truth poses accurate within a few millimeters. We also propose a new pose evaluation metric called ADD-H based on the Hungarian assignment algorithm that is robust to symmetries in object geometry without requiring their explicit enumeration. We share pre-trained pose estimators for all the toy grocery objects, along with their baseline performance on both validation and test sets. We offer this dataset to the community to help connect the efforts of computer vision researchers with the needs of roboticists. (Source: https://github.com/swtyree/hope-dataset)
Introduction
In the domain of robotic grasping, 6-DoF (degrees of freedom) pose estimation is a fundamental task: a robot must determine the exact position and orientation of an object to manipulate it successfully. However, the field currently faces a reproducibility crisis. Popular datasets like YCB or RU-APC often feature real-world grocery items that undergo frequent packaging or branding changes, making it difficult for researchers to source identical items for physical experiments. Furthermore, many existing dataset items are either too heavy, large, or fragile for standard robotic grippers.
NVIDIA researchers have addressed these challenges by releasing the HOPE (Household Objects for Pose Estimation) dataset. This collection features 28 toy grocery objects designed to be globally accessible, durable, and perfectly sized for robotic manipulation.
The HOPE Dataset
The authors curated 28 toy grocery items—including cans, boxes, and bottles—from the "Melissa & Doug" brand. By choosing toy objects available through major global retailers, the researchers ensure that any lab can replicate their results on a physical robot for a minimal cost (less than $60).
To support modern machine learning workflows, the team employed several key techniques:
- 3D Scanning & Refinement: Objects were scanned using an industrial desktop scanner. Raw scans were refined in Maya to create high-fidelity textured meshes suitable for photorealistic synthetic data generation.
- Semi-Automated Annotation: Real-world images were labeled using a tool that identifies correspondences between 2D images and 3D models. These poses were automatically refined using SimTrack, resulting in ground-truth accuracy within a few millimeters. During this process, the team discovered and corrected a systematic 2% scaling error in the Intel RealSense D415 depth sensor, which was crucial for achieving high precision.
The ADD-H Metric
Evaluating symmetric objects, such as a soup can, is notoriously difficult. Standard metrics often penalize "correct" rotations that appear identical (ADD) or are too lenient (ADD-S). The authors propose ADD-H, which utilizes the Hungarian assignment algorithm to find a bijective mapping between model vertices. This allows the metric to be robust to symmetries without requiring researchers to manually enumerate every possible symmetry for every object.
Dataset Characteristics and Benchmarking
The HOPE dataset includes 238 RGB-D images across 50 unique scenes, providing 914 labeled object poses. These scenes capture challenging environments like kitchens and offices, featuring extreme lighting, heavy occlusion, and multiple instances of the same object.
The authors benchmarked state-of-the-art estimators, DOPE and CosyPose, on the HOPE dataset. Key findings include:
- Refinement Impact: Even simple depth-based refinement significantly improved the performance of RGB-only estimators. CosyPose-LS emerged as the top performer, achieving an average recall ($AR_{MSSD}$) of 0.594 on the test set.
- Graspability: At a 10cm error threshold (suitable for coarse detection), the system detected 83% of objects. At a 2cm threshold (the precision typically needed for a successful grasp), it maintained a 72% success rate.
- Lighting Robustness: The models showed resilience to lighting changes, with only minor performance dips between favorable and difficult lighting conditions.
Conclusion and Future Work
The HOPE dataset provides a bridge between computer vision benchmarks and physical robotics. While current estimators perform well, nearly 20% of objects remain undetected at a 10cm threshold in high-clutter scenarios, suggesting significant room for algorithmic improvement. Future work may involve larger-scale automated capture to expand the real-world image count.
Summary updated