Self-supervised Wide Baseline Visual Servoing via 3D Equivariance

Jinwook Huh, Jungseok Hong, Suveer Garg, Hyun Soo Park, Volkan Isler

Introduction

Visual servoing is a cornerstone of robotic manipulation, enabling a robot to use real-time camera feedback to navigate toward a specific goal configuration. Traditionally, this is achieved through Image-Based Visual Servoing (IBVS), which relies on matching local features (e.g., SIFT) between current and target views. However, these methods often fail in "wide baseline" scenarios—where the starting position and the goal are significantly different—because local features are lost due to perspective shifts or occlusions.

While deep learning has offered a potential solution by regressing 3D poses directly, these models typically require extensive 3D supervision, such as CAD models or precise 3D bounding boxes. This paper introduces a self-supervised alternative that bridges the gap: a method capable of handling wide baselines by learning from camera motion (egomotion) rather than expensive 3D labels.

Methodology

The proposed approach utilizes a Siamese network architecture designed to learn a structured visual representation by enforcing two primary geometric constraints: 3D equivariance and geodesic preservation.

3D Equivariance

The core principle is that a 3D transformation applied to an object should result in a predictable, mathematical change in its visual feature representation. The system employs a feature extractor ($f$) and a feature transformer ($h$). For a given image $I$ and a transformation $g$, the network is trained such that the transformed features of the original image match the features of the image after the transformation has occurred. This ensures the latent space captures the underlying geometry of the scene.

Geodesic Preservation

To ensure the learned feature space is well-behaved and avoid trivial solutions (where the network might map all inputs to a single point), the authors introduce a geodesic-preserving constraint. This requires the distance between two points in the feature space to be proportional to the actual 3D geodesic distance of the camera's transformation. This creates a "map-like" latent space where physical movement corresponds directly to movement within the representation.

Implementation and Inference

To stabilize training, the system employs an "object centering" technique, cropping and scaling images so the target object remains centered. This reduces the search space from six degrees of freedom (6-DOF) to three (rotation), significantly improving convergence. At inference time, the robot performs visual servoing by executing gradient descent within this learned feature space, iteratively adjusting its pose to minimize the discrepancy between its current features and the target features.

Experimental Results

The model was trained using 140,000 rendered images of objects from the YCB dataset, a standard benchmark for robotic manipulation. Evaluation was conducted across 2.8 million image pairs to assess the accuracy of relative transformations.

Performance Metrics

Compared to traditional IBVS and supervised methods like PoseCNN, the proposed self-supervised method demonstrated superior robustness:

  • Accuracy: The method achieved a 35% reduction in average distance error (ADD) compared to PoseCNN, despite PoseCNN requiring 3D ground truth during training.
  • Success Rate: The system maintained a success rate of over 90% within a 3cm error tolerance for the majority of tested objects.
  • Real-World Deployment: The approach was validated on a physical robotic arm using an Intel RealSense camera. In real-world tests, the system achieved final pose errors of less than 5cm and 5 degrees, confirming its practical utility.

Discussion and Conclusion

This research demonstrates that camera egomotion—a signal readily available to most robotic systems—is a sufficient supervisory signal for learning complex visual servoing tasks. By structuring the latent space through equivariance and geodesic constraints, the model generalizes better than direct regression methods and handles wide baselines more effectively than local feature matching.

Currently, the system assumes the target object is mostly visible and performs best under mild occlusion. Future work will focus on extending this framework to more cluttered environments and multi-object scenes, eventually moving beyond the requirement for object centering to handle the full complexity of unstructured real-world settings.

Summary updated

Abstract

One of the challenging input settings for visual servoing is when the initial and goal camera views are far apart. Such settings are difficult because the wide baseline can cause drastic changes in object appearance and cause occlusions. This paper presents a novel self-supervised visual servoing method for wide baseline images which does not require 3D ground truth supervision. Existing approaches that regress absolute camera pose with respect to an object require 3D ground truth data of the object in the forms of 3D bounding boxes or meshes. We learn a coherent visual representation by leveraging a geometric property called 3D equivariance—the representation is transformed in a predictable way as a function of 3D transformation. To ensure that the feature-space is faithful to the underlying geodesic space, a geodesic preserving constraint is applied in conjunction with the equivariance. We design a Siamese network that can effectively enforce these two geometric properties without requiring 3D supervision. With the learned model, the relative transformation can be inferred simply by following the gradient in the learned space and used as feedback for closed-loop visual servoing. Our method is evaluated on objects from the YCB dataset, showing meaningful outperformance on a visual servoing task, or object alignment task with respect to state-of-the-art approaches that use 3D supervision. Ours yields more than 35% average distance error reduction and more than 90% success rate with 3cm error tolerance.
BibTeX
@inproceedings{huh2022self,
  author = {Jinwook Huh and Jungseok Hong and Suveer Garg and Hyun Soo Park and Volkan Isler},
  title = {{Self-supervised Wide Baseline Visual Servoing via 3D Equivariance}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=fseMGxqkgtl},
}
Figure