Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation

Hoang-Giang Cao, Weihao Zeng, I-Chen Wu

TL;DR

SRDONs (Sim-to-Real Dense Object Nets) bridge the gap between simulation and the real world by learning dense object descriptors that remain consistent across both domains. By using 3D models to anchor pixel-to-pixel correspondences, the researchers enabled robots to perform complex manipulation tasks—like grasping and picking in cluttered scenes—with zero real-world training.

What Problem Does This Paper Address?

In robotics, "Sim-to-Real" transfer is the holy grail: training an AI in a safe, fast simulation and having it work perfectly on a physical robot. However, two major hurdles remain. First, robots need to "understand" objects beyond simple bounding boxes; they need dense descriptors (coordinate-like maps) that capture the detailed geometry of an object. Second, there is a massive visual gap—simulated images rarely look exactly like real-world ones.

Existing methods like Dense Object Nets (DONs) are great at learning these descriptors but usually require labor-intensive real-world data collection or only work within a single domain. SRDONs aim to solve this by creating a unified feature space where a specific point on a banana in a simulation has the same "fingerprint" (descriptor) as that same point on a real banana.

Methods

The core innovation is a novel Object-to-Object Matching method.

  • 3D-Anchored Correspondence: Instead of just comparing two images of the same scene, the authors use 3D meshes and object poses. They project 3D vertices from a model onto 2D images from different scenes and different domains (Sim vs. Real). This allows the network to know that "Pixel A in Sim" and "Pixel B in Real" represent the exact same physical point on the object.
  • Contrastive Learning: The system uses a contrastive loss function to "pull" matching points together in the feature space and "push" non-matching points apart.
  • Texture Randomization: To prevent the robot from over-relying on specific colors or lighting, they use heavy texture randomization in simulation. This forces the model to focus on the underlying geometry and shape.
  • Architecture: The team used a U-Net-based architecture to generate these descriptors and integrated them into a reinforcement learning (PPO) framework for task execution.

Datasets

  • GraspNet-1billion: A massive public dataset used for real-world training data, providing nearly 100,000 RGBD images of 88 different objects.
  • V-REP Simulator: Used to generate the synthetic counterpart data, where the researchers could perfectly control object poses and meshes to create ground-truth matches.
  • Novel Objects: For evaluation, the team tested the system on household objects never seen during the training phase to prove generalization.

Key Insights and Ideas

  • Geometry is King: By using texture randomization, the authors found that the descriptors become much more robust. The network learns to "see" the shape of the object rather than just its color, which is vital for sim-to-real transfer where lighting and textures often differ.
  • Leveraging Public Data: Instead of spending weeks recording a robot in a lab, the authors showed that you can use existing large-scale datasets (like GraspNet) combined with simulation to achieve high-quality results.
  • Pixel Consistency: The "aha moment" is that if you can map a simulated pixel and a real pixel to the same point in a high-dimensional feature space, the robot’s "brain" (the policy) doesn't actually need to know if it's looking at a simulation or reality.

Key Results

  • Superior Matching: In sim-to-real matching tests, SRDONs achieved 91-94% accuracy, while the original DONs (trained only on real data) plummeted to around 14% when asked to bridge the sim-to-real gap.
  • Zero-Shot Grasping: A robot trained entirely in simulation achieved a 90% success rate when grasping objects in the real world.
  • Cluttered Environment Picking: In a complex "picking" task involving suctioning objects out of a pile, SRDONs achieved a 100% completion rate in the real world, significantly outperforming standard domain randomization techniques.

Limitations and Future Work

While impressive, the method currently relies on having access to 3D models and pose annotations for the training objects. While these are available in many public datasets, it might be a bottleneck for entirely new, custom objects where a 3D scan isn't available.

The authors suggest that future work could focus on further accelerating the deployment process and perhaps finding ways to generate these descriptors without requiring explicit 3D meshes, potentially moving toward even more "in-the-wild" robotic learning.

Summary updated

Abstract

It is crucial to address the following issues for ubiquitous robotics manipulation applications: (a) vision-based manipulation tasks require the robot to visually learn and understand the object with rich information like dense object descriptors; and (b) sim-to-real transfer in robotics aims to close the gap between simulated and real data. In this paper, we present Sim-to-Real Dense Object Nets (SRDONs), a dense object descriptors that not only understands the object via appropriate representation but also maps simulated and real data to a unified feature space with pixel consistency. We proposed an object-to-object matching method for image pairs from different scenes and different domains. This method helps reduce the effort of training data from real-world by taking advantage of public datasets, such as GraspNet. With sim-to-real object representation consistency, our SRDONs can serve as a building block for a variety of sim-to-real manipulation tasks. We demonstrate in experiments that pre-trained SRDONs significantly improve performances on unseen objects and unseen visual environments for various robotic tasks with zero real-world training.
BibTeX
@inproceedings{cao2023learning,
  author = {Hoang-Giang Cao and Weihao Zeng and I-Chen Wu},
  title = {{Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10161477},
}

Loading related content...

Figure