Introduction
Visual servoing is a cornerstone of robotic manipulation, enabling a robot to use real-time camera feedback to navigate toward a specific goal configuration. Traditionally, this is achieved through Image-Based Visual Servoing (IBVS), which relies on matching local features (e.g., SIFT) between current and target views. However, these methods often fail in "wide baseline" scenarios—where the starting position and the goal are significantly different—because local features are lost due to perspective shifts or occlusions.
While deep learning has offered a potential solution by regressing 3D poses directly, these models typically require extensive 3D supervision, such as CAD models or precise 3D bounding boxes. This paper introduces a self-supervised alternative that bridges the gap: a method capable of handling wide baselines by learning from camera motion (egomotion) rather than expensive 3D labels.
Methodology
The proposed approach utilizes a Siamese network architecture designed to learn a structured visual representation by enforcing two primary geometric constraints: 3D equivariance and geodesic preservation.
3D Equivariance
The core principle is that a 3D transformation applied to an object should result in a predictable, mathematical change in its visual feature representation. The system employs a feature extractor ($f$) and a feature transformer ($h$). For a given image $I$ and a transformation $g$, the network is trained such that the transformed features of the original image match the features of the image after the transformation has occurred. This ensures the latent space captures the underlying geometry of the scene.
Geodesic Preservation
To ensure the learned feature space is well-behaved and avoid trivial solutions (where the network might map all inputs to a single point), the authors introduce a geodesic-preserving constraint. This requires the distance between two points in the feature space to be proportional to the actual 3D geodesic distance of the camera's transformation. This creates a "map-like" latent space where physical movement corresponds directly to movement within the representation.
Implementation and Inference
To stabilize training, the system employs an "object centering" technique, cropping and scaling images so the target object remains centered. This reduces the search space from six degrees of freedom (6-DOF) to three (rotation), significantly improving convergence. At inference time, the robot performs visual servoing by executing gradient descent within this learned feature space, iteratively adjusting its pose to minimize the discrepancy between its current features and the target features.
Experimental Results
The model was trained using 140,000 rendered images of objects from the YCB dataset, a standard benchmark for robotic manipulation. Evaluation was conducted across 2.8 million image pairs to assess the accuracy of relative transformations.
Performance Metrics
Compared to traditional IBVS and supervised methods like PoseCNN, the proposed self-supervised method demonstrated superior robustness:
- Accuracy: The method achieved a 35% reduction in average distance error (ADD) compared to PoseCNN, despite PoseCNN requiring 3D ground truth during training.
- Success Rate: The system maintained a success rate of over 90% within a 3cm error tolerance for the majority of tested objects.
- Real-World Deployment: The approach was validated on a physical robotic arm using an Intel RealSense camera. In real-world tests, the system achieved final pose errors of less than 5cm and 5 degrees, confirming its practical utility.
Discussion and Conclusion
This research demonstrates that camera egomotion—a signal readily available to most robotic systems—is a sufficient supervisory signal for learning complex visual servoing tasks. By structuring the latent space through equivariance and geodesic constraints, the model generalizes better than direct regression methods and handles wide baselines more effectively than local feature matching.
Currently, the system assumes the target object is mostly visible and performs best under mild occlusion. Future work will focus on extending this framework to more cluttered environments and multi-object scenes, eventually moving beyond the requirement for object centering to handle the full complexity of unstructured real-world settings.
Summary updated