Abstract
Accurate 3D pose estimation is critical for robotic manipulation, yet achieving sub-centimeter precision with standard RGB cameras remains challenging. This paper presents Inverse Transform based Domain Randomization (ITDR), an active perception framework that bridges the sim-to-real gap. By leveraging known robotic movements to aggregate multiple viewpoints, ITDR reduces pose estimation error from approximately 1.5cm to under 0.5cm. Our results demonstrate that geometric consistency across active viewpoints allows for high-precision industrial tasks without the need for expensive depth sensors or fiducial markers.
1. Introduction
In robotic manipulation, knowing the precise 3D state of an object is fundamental. While specialized hardware like depth cameras or fiducial markers (e.g., AprilTags) can provide this information, they are often unsuitable for cluttered industrial environments or small-batch production where setup time is limited. Domain Randomization (DR) has emerged as a powerful technique to train neural networks in simulation for real-world deployment. However, traditional passive DR methods typically hit a "precision ceiling" of approximately 1.5cm, which is insufficient for high-precision tasks like peg-in-hole insertion or complex assembly.
This work addresses this limitation by introducing Active Perception into the DR pipeline. Instead of relying on a single static image, the robot actively interacts with the scene—moving itself or the object—to gather more information and refine its spatial understanding.
2. The ITDR Framework
The proposed method, Inverse Transform based Domain Randomization (ITDR), consists of a simulation-to-real training phase followed by an active inference phase.
2.1 Training via Domain Randomization
A VGG-style Convolutional Neural Network (CNN) is trained entirely within a simulated environment. To ensure the model generalizes to the real world, we employ heavy domain randomization. By varying textures, lighting, and backgrounds while keeping the object geometry constant, the network is forced to learn robust geometric features rather than relying on superficial visual cues.
2.2 Active Inference and Aggregation
At test time, the robot captures multiple images of the target object. Between captures, the robot performs a known geometric transformation $T$ (e.g., a 5cm translation or a 30-degree rotation). The CNN predicts a 3D pose for each viewpoint. Because the relative transformation between viewpoints is known from the robot's kinematics, we can apply the inverse transformation ($T^{-1}$) to align all predictions into a common coordinate frame.
The final pose estimate is the average of these aligned predictions. This process acts as a geometric consistency filter, effectively canceling out the stochastic noise present in individual single-view predictions.
3. Experimental Evaluation
The system was evaluated using a real-world robotic arm setup. Training data was generated using a custom simulation environment featuring peg-shaped objects and reference cylinders. The evaluation focused on three primary scenarios:
- Moving Reference Objects: Estimating the pose of objects on a moving platform.
- In-Hand Manipulation: Estimating the pose of an object held by the robot arm.
- Eye-in-Hand: Estimating object pose using a camera mounted on the robot's end-effector.
4. Results and Discussion
The ITDR method demonstrated substantial improvements in both translation and rotation accuracy across all test scenarios.
- Translation Precision: In the reference object task, the mean error dropped from 1.57cm using a single image to 0.62cm when aggregating four images.
- Rotation Accuracy: For in-hand tasks, angular error was reduced by nearly 3x, from 0.129 to 0.047 radians.
- Robustness to Occlusion: Active perception allows the robot to overcome self-occlusion or ambiguous viewpoints (e.g., head-on views of asymmetric objects) by "peeking" from a different angle to find a clearer view.
The system ultimately achieved sub-5mm accuracy. This level of precision is a significant milestone for DR-based methods, making them viable for high-precision industrial assembly tasks that previously required expensive 3D sensors.
5. Conclusion and Future Work
ITDR demonstrates that a robot's own movement can serve as a powerful source of "ground truth" information to refine visual perception. By enforcing geometric consistency across multiple views, high precision can be achieved using only low-cost RGB sensors.
Current limitations include the requirement for accurate 3D CAD models for simulation and the assumption of rigid body transformations; any slippage in the gripper during the active phase can degrade performance. Future research will explore semi-supervised learning techniques to fine-tune simulation-trained models on unlabeled real-world data, potentially further increasing precision and reducing the reliance on perfect CAD models.
Summary updated