Markerless Camera-to-Robot Pose Estimation via Self-Supervised Sim-to-Real Transfer

Jingpei Lu, Florian Richter, Michael C. Yip

Abstract

Precise camera-to-robot pose estimation is a fundamental requirement for robotic manipulation tasks. Traditional methods rely on fiducial markers or extensive manual labeling, which are difficult to scale. We present CtRNet, a markerless framework that achieves state-of-the-art accuracy at real-time speeds (30 FPS). By integrating a differentiable renderer into a self-supervised training loop, CtRNet bridges the "sim-to-real" gap, allowing models trained in simulation to adapt to real-world environments without manual 3D annotations.

Introduction

For robots to interact accurately with their surroundings—whether in surgical settings or industrial bin-picking—knowing their exact pose relative to the camera is essential. This "camera-to-robot" pose estimation has traditionally depended on fiducial markers like Aruco tags. However, markers are prone to falling off, require manual recalibration if shifted, and can be obscured.

While deep learning offers a markerless alternative, models trained on synthetic data often fail in the real world due to the "sim-to-real gap"—differences in lighting, texture, and background. Obtaining the 3D labels necessary to train these models on real-world data is prohibitively expensive. CtRNet addresses this by enabling self-supervised fine-tuning on unlabeled real-world images.

Methodology

The CtRNet architecture utilizes a ResNet-50 backbone that branches into two parallel tasks: foreground segmentation and 2D keypoint detection.

Geometric Reasoning and Differentiable PnP

The network detects 2D keypoints in the image, which are then paired with 3D keypoints from the robot's internal kinematics. A Perspective-n-Point (PnP) solver calculates the 6-DOF pose. A key technical contribution is making this PnP step fully differentiable using the implicit function theorem, allowing gradients to propagate through the entire pipeline during training.

Self-Supervised Sim-to-Real Transfer

The core innovation of CtRNet is its self-supervised feedback loop. After initial pre-training on synthetic data, the model processes real RGB images to estimate the robot's pose.

The self-supervised training loop of CtRNet, utilizing a differentiable renderer to bridge the sim-to-real gap.

As shown in the figure above, a differentiable renderer generates a 2D silhouette of the robot based on the estimated pose. This rendered silhouette is compared against the output of the network's own segmentation module. The resulting "image-level" loss allows the network to refine its keypoint detection and segmentation capabilities simultaneously. This creates a "mutual supervision" effect where the segmentation mask and the renderer provide geometric sanity checks for each other. To ensure stability, the authors also implemented a weighted loss mechanism based on reprojection error, preventing noisy pose estimates from negatively impacting the training process.

Experimental Results

The researchers evaluated CtRNet on two major real-world benchmarks: the DREAM-real Dataset (~50,000 images of a Franka Emika Panda arm) and the Baxter Dataset.

Performance and Efficiency

CtRNet achieves a significant performance leap over existing methods. On the DREAM-real dataset, it reached an 85.9% AUC (Area Under Curve for the ADD metric). This outperforms previous keypoint-based methods like DREAM (~68%) and even exceeds the accuracy of slower, rendering-based methods like RoboPose (~80%).

Comparison of CtRNet against state-of-the-art methods in terms of accuracy (AUC) and speed (FPS).

Crucially, CtRNet maintains a processing speed of 30 FPS, making it suitable for real-time closed-loop control. In practical tests using Position-Based Visual Servoing (PBVS), the system achieved a translational error of just 2mm, demonstrating its robustness for high-precision tasks.

Discussion and Conclusion

CtRNet effectively balances the speed of keypoint-based methods with the accuracy of rendering-based approaches by moving the heavy rendering computation to the training phase. While the system currently faces challenges in extremely cluttered environments or when the robot is partially out of the camera's field of view, it represents a major step toward "plug-and-play" robotic vision. Future work will explore extending this self-supervised approach to multi-robot systems and more unstructured environments.

Summary updated

Abstract

Solving the camera-to-robot pose is a fundamental requirement for vision-based robot control, and is a process that takes considerable effort and cares to make accurate. Traditional approaches require modification of the robot via markers, and subsequent deep learning approaches enabled markerless feature extraction. Mainstream deep learning methods only use synthetic data and rely on Domain Randomization to fill the sim-to-real gap, because acquiring the 3D annotation is labor-intensive. In this work, we go beyond the limitation of 3D annotations for real-world data. We propose an end-to-end pose estimation framework that is capable of online camera-to-robot calibration and a self-supervised training method to scale the training to unlabeled real-world data. Our framework combines deep learning and geometric vision for solving the robot pose, and the pipeline is fully differentiable. To train the Camera-to-Robot Pose Estimation Network (CtRNet), we leverage foreground segmentation and differentiable rendering for image-level self-supervision. The pose prediction is visualized through a renderer and the image loss with the input image is back-propagated to train the neural network. Our experimental results on two public real datasets confirm the effectiveness of our approach over existing works. We also integrate our framework into a visual servoing system to demonstrate the promise of real-time precise robot pose estimation for automation tasks.
BibTeX
@inproceedings{lu2023markerless,
  author = {Jingpei Lu and Florian Richter and Michael C. Yip},
  title = {{Markerless Camera-to-Robot Pose Estimation via Self-Supervised Sim-to-Real Transfer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02040},
}
Figure