Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation

Hoang-Giang Cao, Weihao Zeng, I-Chen Wu

Abstract

It is crucial to address the following issues for ubiquitous robotics manipulation applications: (a) vision-based manipulation tasks require the robot to visually learn and understand the object with rich information like dense object descriptors; and (b) sim-to-real transfer in robotics aims to close the gap between simulated and real data. In this paper, we present Sim-to-Real Dense Object Nets (SRDONs), a dense object descriptor that not only understands the object via appropriate representation but also maps simulated and real data to a unified feature space with pixel consistency. We proposed an object-to-object matching method for image pairs from different scenes and different domains. This method helps reduce the effort of training data from real-world by taking advantage of public datasets, such as GraspNet. With sim-to-real object representation consistency, our SRDONs can serve as a building block for a variety of sim-to-real manipulation tasks. We demonstrate in experiments that pre-trained SRDONs significantly improve performances on unseen objects and unseen visual environments for various robotic tasks with zero real-world training.

Introduction

In robotics, "Sim-to-Real" transfer—the process of training an agent in a simulated environment and deploying it in the physical world—remains a significant challenge. Two major hurdles impede this transition. First, robots require a deep understanding of objects beyond simple bounding boxes; they need dense descriptors (coordinate-like maps) that capture detailed geometry. Second, a substantial visual gap exists between simulated and real-world images. Existing methods like Dense Object Nets (DONs) often require labor-intensive real-world data collection or fail to generalize across domains. Sim-to-Real Dense Object Nets (SRDONs) address these issues by creating a unified feature space where specific points on an object share the same descriptor "fingerprint" regardless of whether the image is simulated or real.

Methodology

The core innovation of SRDONs is a novel Object-to-Object Matching method that anchors pixel-to-pixel correspondences using 3D models.

3D-anchored correspondence established by projecting 3D models onto real and simulated images based on object pose.

3D-Anchored Correspondence

Instead of comparing images of the same scene, the authors project 3D vertices from a model onto 2D images from different scenes and domains (Sim vs. Real). This allows the network to identify that a pixel in a simulation and a pixel in a real image represent the exact same physical point on an object.

Training and Architecture

The system utilizes a contrastive loss function to align matching points in the feature space while separating non-matching points. To ensure robustness against lighting and color variations, heavy texture randomization is applied in simulation, forcing the model to prioritize underlying geometry. The architecture is based on a U-Net, integrated into a Proximal Policy Optimization (PPO) framework for task execution.

Experiments and Datasets

The model was trained using GraspNet-1billion, a public dataset providing nearly 100,000 RGBD images of 88 objects, and the V-REP Simulator for generating synthetic counterparts with perfect ground-truth matches. Evaluation was conducted on novel household objects never seen during the training phase to test generalization capabilities.

Results and Discussion

SRDONs demonstrate superior performance in bridging the sim-to-real gap. In matching tests, the system achieved 91-94% accuracy, whereas standard DONs trained only on real data dropped to 14% when tested across domains.

Comparison of matching performance across Real-Real, Sim-Sim, and Sim-Real scenarios.

Key findings include:

  • Zero-Shot Grasping: A robot trained entirely in simulation achieved a 90% success rate on real-world objects.
  • Cluttered Environments: In complex picking tasks involving suctioning objects out of a pile, SRDONs reached a 100% completion rate, significantly outperforming standard domain randomization techniques.
  • Geometry Robustness: Texture randomization proved vital, as it compelled the network to learn shape-based descriptors rather than color-dependent ones.

Conclusion

SRDONs provide a robust framework for sim-to-real transfer by leveraging 3D models and public datasets to create consistent dense descriptors. While the current reliance on 3D models and pose annotations for training objects is a potential bottleneck, the method significantly reduces the need for real-world data collection. Future work may explore generating these descriptors without explicit 3D meshes to further streamline "in-the-wild" robotic learning.

Summary updated

Abstract

It is crucial to address the following issues for ubiquitous robotics manipulation applications: (a) vision-based manipulation tasks require the robot to visually learn and understand the object with rich information like dense object descriptors; and (b) sim-to-real transfer in robotics aims to close the gap between simulated and real data. In this paper, we present Sim-to-Real Dense Object Nets (SRDONs), a dense object descriptor that not only understands the object via appropriate representation but also maps simulated and real data to a unified feature space with pixel consistency. We proposed an object-to-object matching method for image pairs from different scenes and different domains. This method helps reduce the effort of training data from real-world by taking advantage of public datasets, such as GraspNet. With sim-to-real object representation consistency, our SRDONs can serve as a building block for a variety of sim-to-real manipulation tasks. We demonstrate in experiments that pre-trained SRDONs significantly improve performances on unseen objects and unseen visual environments for various robotic tasks with zero real-world training.
BibTeX
@inproceedings{cao2023learning,
  author = {Hoang-Giang Cao and Weihao Zeng and I-Chen Wu},
  title = {{Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10161477},
}
Figure