Self-Supervised Geometric Perception

Heng Yang, Wei Dong, Luca Carlone, Vladlen Koltun

Introduction

Geometric perception—the task of estimating camera poses or 3D structures from visual data—is a cornerstone of computer vision. While deep learning has significantly advanced the "front-end" of these systems (feature detection and description), training these models typically requires massive amounts of ground-truth data, such as exact camera coordinates or manually-tuned 3D reconstructions. Obtaining such labels is often prohibitively expensive or technically challenging.

Presented at CVPR 2021, this research introduces Self-supervised Geometric Perception (SGP), a framework that learns high-quality feature descriptors without any ground-truth labels. By framing geometric perception as a joint optimization problem, SGP enables a neural network to learn from a robust mathematical "teacher."

The SGP Framework

The core innovation of SGP is treating feature learning and model fitting as a single, unified optimization task. The algorithm employs an alternating minimization strategy through an iterative teacher-student loop.

Overview of the SGP framework showing the iterative teacher-student loop between the robust estimator (Teacher) and the neural network (Student).

The Iterative Loop

  1. The Teacher (Robust Estimation): Using current feature descriptors, a robust algorithm (such as RANSAC) estimates the best geometric model. These estimates serve as "pseudo-labels."
  2. The Verifier (Quality Control): To prevent the network from learning from incorrect estimates, a verifier assesses the quality of the pseudo-labels (e.g., via inlier rates). Low-quality data is discarded.
  3. The Student (Deep Feature Learning): A neural network is trained to produce improved descriptors using the verified pseudo-labels as its target.
  4. Bootstrapping: The process begins with simple, hand-crafted features (like SIFT or FPFH) to generate the initial set of pseudo-labels, allowing the system to start from scratch.

Experimental Evaluation

The authors evaluated SGP across diverse benchmarks to test its ability to generalize across different geometric tasks and environments.

  • MegaDepth: Used for relative camera pose estimation from Internet images.
  • 3DMatch: A benchmark for 3D point cloud registration in indoor environments.
  • Generalization: The models were further validated on unseen datasets like ScanNet and the Stanford RGBD dataset to ensure the learned features were not environment-specific.

Key Results and Insights

The SGP framework demonstrated that robust model fitting and supervised feature learning are theoretically two sides of the same coin. This mathematical unity allows the system to create a "virtuous cycle": as the student learns better features, the teacher finds more accurate inliers, which in turn provides even better training data.

Qualitative examples of learned correspondences across various indoor and outdoor scenes, demonstrating the high density and accuracy of the SGP features.

Key performance highlights include:

  • Relative Pose (MegaDepth): The self-supervised descriptor, S-CAPS, significantly outperformed traditional SIFT and matched the performance of supervised versions of the same architecture.
  • Point Cloud Registration (3DMatch): The self-supervised model (S-FCGF) achieved a 91.4% recall, slightly exceeding the supervised "Oracle" model (91.1%) and outperforming other state-of-the-art methods.
  • Cross-Dataset Transfer: Features learned on one dataset transferred effectively to others, proving that SGP learns fundamental geometric principles rather than just memorizing specific data distributions.

Conclusion and Future Work

SGP proves that ground-truth labels are not a prerequisite for high-performance geometric perception. By leveraging robust estimation as a teacher, the framework reaches "Oracle" levels of accuracy. Future research may focus on increasing the convergence speed of the loop and exploring fully differentiable robust estimation layers to further integrate the teacher and student components.

Summary updated

Abstract

We present self-supervised geometric perception (SGP), the first general framework to learn a feature descriptor for correspondence matching without any ground-truth geometric model labels (e.g., camera poses, rigid transformations). Our first contribution is to formulate geometric perception as an optimization problem that jointly optimizes the feature descriptor and the geometric models given a large corpus of visual measurements (e.g., images, point clouds). Under this optimization formulation, we show that two important streams of research in vision, namely robust model fitting and deep feature learning, correspond to optimizing one block of the unknown variables while fixing the other block. This analysis naturally leads to our second contribution -- the SGP algorithm that performs alternating minimization to solve the joint optimization. SGP iteratively executes two meta-algorithms: a teacher that performs robust model fitting given learned features to generate geometric pseudo-labels, and a student that performs deep feature learning under noisy supervision of the pseudo-labels. As a third contribution, we apply SGP to two perception problems on large-scale real datasets, namely relative camera pose estimation on MegaDepth and point cloud registration on 3DMatch. We demonstrate that SGP achieves state-of-the-art performance that is on-par or superior to the supervised oracles trained using ground-truth labels.
BibTeX
@inproceedings{yang2021self,
  author = {Heng Yang and Wei Dong and Luca Carlone and Vladlen Koltun},
  title = {{Self-Supervised Geometric Perception}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.01412},
}
Figure