TL;DR
This paper introduces EasyHeC++, a novel framework for fully automatic hand-eye calibration that eliminates the need for manual calibration, specialized markers, or training robot-specific neural networks. By combining pre-trained image models with differentiable rendering, EasyHeC++ achieves high accuracy and significantly reduces setup time for both eye-to-hand and eye-in-hand configurations.
What Problem Does This Paper Address?
Hand-eye calibration is a critical step in robotics, enabling robots to accurately perceive and interact with their environment. It establishes the spatial relationship between a robot's end-effector and its camera. Traditional methods often rely on markers and manual pose selection, which are cumbersome and time-consuming. More recent learning-based approaches have reduced the need for markers but still require extensive training data and models specific to each robot arm, hindering their widespread adoption, especially in dynamic or household environments where frequent recalibration might be necessary. This paper addresses the need for a truly automatic, marker-free, and training-free hand-eye calibration system that is both accurate and efficient.
Methods
EasyHeC++ employs a two-stage process:
- Pose Initialization:
- Sampling-based Initialization (for first-time calibration): This method samples camera poses on a hemisphere and uses a differentiable renderer with masks from a text-prompted segmentation model (GroundedSAM) to find an initial camera pose.
- Feature-Matching-based Initialization (for re-calibration): This leverages previously stored image-pose pairs in a database. It uses a pre-trained dense feature-matching network (DKM) to find correspondences between a new image and a reference image, then lifts these to 3D to estimate the camera pose. This approach is automatic and not robot-specific.
- Pose Optimization:
- This stage builds upon the EasyHeC framework, using differentiable rendering with a pixel-wise mask loss to refine the camera pose.
- AutoSAM Module: A key innovation is the AutoSAM module, which uses the robot arm's kinematics model to guide the Segment Anything (SAM) model. This allows for automatic, zero-shot mask generation without manual annotation or robot-specific training, significantly improving automation.
- Consistency-based Joint Space Exploration: This module searches for informative joint poses to further enhance calibration accuracy.
The framework supports both eye-to-hand and eye-in-hand configurations.
Datasets
- Synthetic Datasets: The paper evaluates its method on synthetic datasets for both eye-to-hand and eye-in-hand settings, including a dataset proposed in [19] and another simulated using SAPIEN [41].
- Real-world Datasets:
- Baxter Dataset: Used for evaluating 2D and 3D PCK scores.
- Real-world Setup: Experiments were conducted using a xArm7 robot arm with a RealSense camera.
Key Insights and Ideas
- Leveraging Pre-trained Models for Automation: The core idea is to combine the generalization power of large, pre-trained image models (like SAM) with the precision of differentiable rendering. This bypasses the need for robot-specific training data and models.
- Training-Free Calibration: The system is designed to work without any prior training on robot-specific data, making it universally applicable.
- Automatic Mask Generation: The AutoSAM module, guided by robot kinematics, is a significant step towards fully automatic segmentation for calibration, eliminating manual annotation.
- Efficient Re-calibration: The feature-matching-based initialization allows for rapid re-calibration after camera movement, a crucial feature for practical deployment.
- Prompt Engineering for SAM: The ablation study highlights the importance of sophisticated prompting strategies (e.g., per-link bounding boxes with connectors) for SAM to achieve high-quality masks for complex articulated objects like robot arms.
Key Results
- Accuracy: EasyHeC++ achieves superior accuracy compared to previous methods, with reported translation errors as low as 0.135cm on synthetic datasets and 0.3cm on real-world experiments.
- Automation: It is the first system to achieve fully automatic hand-eye calibration without markers, training, or manual annotation.
- Speed: Setup time for a new robot arm is approximately 15 minutes, and re-calibration takes about 5 minutes.
- Generalization: The method demonstrates strong performance across various robot arms and camera settings in both synthetic and real-world evaluations.
- PCK Scores: On the Baxter dataset, EasyHeC++ achieved significantly higher 2D and 3D PCK scores compared to prior methods, especially under stricter thresholds.
Limitations and Future Work
- Limitations:
- While the sampling-based initialization is robust to mask quality, it can be time-consuming (around 10 minutes) as it requires moving the robot arm to multiple poses.
- The paper notes that for SAM, using more complex prompts (e.g., per-link bounding boxes with connectors) yields better mask quality but requires careful configuration.
- Future Work: The paper suggests that this work opens up possibilities for applications requiring hand-eye calibration to reduce the sim-to-real gap, such as robot manipulation and grasping in household or dynamic environments. The authors are also committed to open-sourcing their system.
Summary updated