Domain Randomization for Active Pose Estimation.

Xinyi Ren, Jianlan Luo, Eugen Solowjow, Juan Aparicio Ojea, Abhishek Gupta, Aviv Tamar, Pieter Abbeel

Abstract

Accurate 3D pose estimation is critical for robotic manipulation, yet achieving sub-centimeter precision with standard RGB cameras remains challenging. This paper presents Inverse Transform based Domain Randomization (ITDR), an active perception framework that bridges the sim-to-real gap. By leveraging known robotic movements to aggregate multiple viewpoints, ITDR reduces pose estimation error from approximately 1.5cm to under 0.5cm. Our results demonstrate that geometric consistency across active viewpoints allows for high-precision industrial tasks without the need for expensive depth sensors or fiducial markers.

1. Introduction

In robotic manipulation, knowing the precise 3D state of an object is fundamental. While specialized hardware like depth cameras or fiducial markers (e.g., AprilTags) can provide this information, they are often unsuitable for cluttered industrial environments or small-batch production where setup time is limited. Domain Randomization (DR) has emerged as a powerful technique to train neural networks in simulation for real-world deployment. However, traditional passive DR methods typically hit a "precision ceiling" of approximately 1.5cm, which is insufficient for high-precision tasks like peg-in-hole insertion or complex assembly.

This work addresses this limitation by introducing Active Perception into the DR pipeline. Instead of relying on a single static image, the robot actively interacts with the scene—moving itself or the object—to gather more information and refine its spatial understanding.

2. The ITDR Framework

The proposed method, Inverse Transform based Domain Randomization (ITDR), consists of a simulation-to-real training phase followed by an active inference phase.

2.1 Training via Domain Randomization

A VGG-style Convolutional Neural Network (CNN) is trained entirely within a simulated environment. To ensure the model generalizes to the real world, we employ heavy domain randomization. By varying textures, lighting, and backgrounds while keeping the object geometry constant, the network is forced to learn robust geometric features rather than relying on superficial visual cues.

Examples of domain randomization in simulation, where textures, lighting, and backgrounds are varied to force the model to learn geometric features.

2.2 Active Inference and Aggregation

At test time, the robot captures multiple images of the target object. Between captures, the robot performs a known geometric transformation $T$ (e.g., a 5cm translation or a 30-degree rotation). The CNN predicts a 3D pose for each viewpoint. Because the relative transformation between viewpoints is known from the robot's kinematics, we can apply the inverse transformation ($T^{-1}$) to align all predictions into a common coordinate frame.

The ITDR pipeline: Multiple viewpoints are aggregated using known geometric transformations to refine pose predictions.

The final pose estimate is the average of these aligned predictions. This process acts as a geometric consistency filter, effectively canceling out the stochastic noise present in individual single-view predictions.

3. Experimental Evaluation

The system was evaluated using a real-world robotic arm setup. Training data was generated using a custom simulation environment featuring peg-shaped objects and reference cylinders. The evaluation focused on three primary scenarios:

  1. Moving Reference Objects: Estimating the pose of objects on a moving platform.
  2. In-Hand Manipulation: Estimating the pose of an object held by the robot arm.
  3. Eye-in-Hand: Estimating object pose using a camera mounted on the robot's end-effector.

4. Results and Discussion

The ITDR method demonstrated substantial improvements in both translation and rotation accuracy across all test scenarios.

  • Translation Precision: In the reference object task, the mean error dropped from 1.57cm using a single image to 0.62cm when aggregating four images.
  • Rotation Accuracy: For in-hand tasks, angular error was reduced by nearly 3x, from 0.129 to 0.047 radians.
  • Robustness to Occlusion: Active perception allows the robot to overcome self-occlusion or ambiguous viewpoints (e.g., head-on views of asymmetric objects) by "peeking" from a different angle to find a clearer view.

The system ultimately achieved sub-5mm accuracy. This level of precision is a significant milestone for DR-based methods, making them viable for high-precision industrial assembly tasks that previously required expensive 3D sensors.

5. Conclusion and Future Work

ITDR demonstrates that a robot's own movement can serve as a powerful source of "ground truth" information to refine visual perception. By enforcing geometric consistency across multiple views, high precision can be achieved using only low-cost RGB sensors.

Current limitations include the requirement for accurate 3D CAD models for simulation and the assumption of rigid body transformations; any slippage in the gripper during the active phase can degrade performance. Future research will explore semi-supervised learning techniques to fine-tune simulation-trained models on unlabeled real-world data, potentially further increasing precision and reducing the reliance on perfect CAD models.

Summary updated

Abstract

Accurate state estimation is a fundamental component of robotic control. In robotic manipulation tasks, as is our focus in this work, state estimation is essential for identifying the positions of objects in the scene, forming the basis of the manipulation plan. However, pose estimation typically requires expensive 3D cameras or additional instrumentation such as fiducial markers to perform accurately. Recently, Tobin et al. introduced an approach to pose estimation based on domain randomization, where a neural network is trained to predict pose directly from a 2D image of the scene. The network is trained on computer generated images with a high variation in textures and lighting, thereby generalizing to real world images. In this work, we investigate how to improve the accuracy of domain randomization based pose estimation. Our main idea is that active perception - moving the robot to get a better estimate of pose- can be trained in simulation and transferred to real using domain randomization. In our approach, the robot trains in a domain-randomized simulation how to estimate pose from a sequence of images. We show that our approach can significantly improve the accuracy of standard pose estimation in several scenarios: when the robot holding an object moves, when reference objects are moved in the scene, or when the camera is moved around the object.
BibTeX
@inproceedings{ren2019domain,
  author = {Xinyi Ren and Jianlan Luo and Eugen Solowjow and Juan Aparicio Ojea and Abhishek Gupta and Aviv Tamar and Pieter Abbeel},
  title = {{Domain Randomization for Active Pose Estimation.}},
  booktitle = {ICRA},
  year = {2019},
  url = {https://openreview.net/forum?id=eHed74dE0g},
}
Figure