Abstract
Precise camera-to-robot pose estimation is a fundamental requirement for robotic manipulation tasks. Traditional methods rely on fiducial markers or extensive manual labeling, which are difficult to scale. We present CtRNet, a markerless framework that achieves state-of-the-art accuracy at real-time speeds (30 FPS). By integrating a differentiable renderer into a self-supervised training loop, CtRNet bridges the "sim-to-real" gap, allowing models trained in simulation to adapt to real-world environments without manual 3D annotations.
Introduction
For robots to interact accurately with their surroundings—whether in surgical settings or industrial bin-picking—knowing their exact pose relative to the camera is essential. This "camera-to-robot" pose estimation has traditionally depended on fiducial markers like Aruco tags. However, markers are prone to falling off, require manual recalibration if shifted, and can be obscured.
While deep learning offers a markerless alternative, models trained on synthetic data often fail in the real world due to the "sim-to-real gap"—differences in lighting, texture, and background. Obtaining the 3D labels necessary to train these models on real-world data is prohibitively expensive. CtRNet addresses this by enabling self-supervised fine-tuning on unlabeled real-world images.
Methodology
The CtRNet architecture utilizes a ResNet-50 backbone that branches into two parallel tasks: foreground segmentation and 2D keypoint detection.
Geometric Reasoning and Differentiable PnP
The network detects 2D keypoints in the image, which are then paired with 3D keypoints from the robot's internal kinematics. A Perspective-n-Point (PnP) solver calculates the 6-DOF pose. A key technical contribution is making this PnP step fully differentiable using the implicit function theorem, allowing gradients to propagate through the entire pipeline during training.
Self-Supervised Sim-to-Real Transfer
The core innovation of CtRNet is its self-supervised feedback loop. After initial pre-training on synthetic data, the model processes real RGB images to estimate the robot's pose.
As shown in the figure above, a differentiable renderer generates a 2D silhouette of the robot based on the estimated pose. This rendered silhouette is compared against the output of the network's own segmentation module. The resulting "image-level" loss allows the network to refine its keypoint detection and segmentation capabilities simultaneously. This creates a "mutual supervision" effect where the segmentation mask and the renderer provide geometric sanity checks for each other. To ensure stability, the authors also implemented a weighted loss mechanism based on reprojection error, preventing noisy pose estimates from negatively impacting the training process.
Experimental Results
The researchers evaluated CtRNet on two major real-world benchmarks: the DREAM-real Dataset (~50,000 images of a Franka Emika Panda arm) and the Baxter Dataset.
Performance and Efficiency
CtRNet achieves a significant performance leap over existing methods. On the DREAM-real dataset, it reached an 85.9% AUC (Area Under Curve for the ADD metric). This outperforms previous keypoint-based methods like DREAM (~68%) and even exceeds the accuracy of slower, rendering-based methods like RoboPose (~80%).
Crucially, CtRNet maintains a processing speed of 30 FPS, making it suitable for real-time closed-loop control. In practical tests using Position-Based Visual Servoing (PBVS), the system achieved a translational error of just 2mm, demonstrating its robustness for high-precision tasks.
Discussion and Conclusion
CtRNet effectively balances the speed of keypoint-based methods with the accuracy of rendering-based approaches by moving the heavy rendering computation to the training phase. While the system currently faces challenges in extremely cluttered environments or when the robot is partially out of the camera's field of view, it represents a major step toward "plug-and-play" robotic vision. Future work will explore extending this self-supervised approach to multi-robot systems and more unstructured environments.
Summary updated