Introduction
Geometric perception—the task of estimating camera poses or 3D structures from visual data—is a cornerstone of computer vision. While deep learning has significantly advanced the "front-end" of these systems (feature detection and description), training these models typically requires massive amounts of ground-truth data, such as exact camera coordinates or manually-tuned 3D reconstructions. Obtaining such labels is often prohibitively expensive or technically challenging.
Presented at CVPR 2021, this research introduces Self-supervised Geometric Perception (SGP), a framework that learns high-quality feature descriptors without any ground-truth labels. By framing geometric perception as a joint optimization problem, SGP enables a neural network to learn from a robust mathematical "teacher."
The SGP Framework
The core innovation of SGP is treating feature learning and model fitting as a single, unified optimization task. The algorithm employs an alternating minimization strategy through an iterative teacher-student loop.
The Iterative Loop
- The Teacher (Robust Estimation): Using current feature descriptors, a robust algorithm (such as RANSAC) estimates the best geometric model. These estimates serve as "pseudo-labels."
- The Verifier (Quality Control): To prevent the network from learning from incorrect estimates, a verifier assesses the quality of the pseudo-labels (e.g., via inlier rates). Low-quality data is discarded.
- The Student (Deep Feature Learning): A neural network is trained to produce improved descriptors using the verified pseudo-labels as its target.
- Bootstrapping: The process begins with simple, hand-crafted features (like SIFT or FPFH) to generate the initial set of pseudo-labels, allowing the system to start from scratch.
Experimental Evaluation
The authors evaluated SGP across diverse benchmarks to test its ability to generalize across different geometric tasks and environments.
- MegaDepth: Used for relative camera pose estimation from Internet images.
- 3DMatch: A benchmark for 3D point cloud registration in indoor environments.
- Generalization: The models were further validated on unseen datasets like ScanNet and the Stanford RGBD dataset to ensure the learned features were not environment-specific.
Key Results and Insights
The SGP framework demonstrated that robust model fitting and supervised feature learning are theoretically two sides of the same coin. This mathematical unity allows the system to create a "virtuous cycle": as the student learns better features, the teacher finds more accurate inliers, which in turn provides even better training data.
Key performance highlights include:
- Relative Pose (MegaDepth): The self-supervised descriptor, S-CAPS, significantly outperformed traditional SIFT and matched the performance of supervised versions of the same architecture.
- Point Cloud Registration (3DMatch): The self-supervised model (S-FCGF) achieved a 91.4% recall, slightly exceeding the supervised "Oracle" model (91.1%) and outperforming other state-of-the-art methods.
- Cross-Dataset Transfer: Features learned on one dataset transferred effectively to others, proving that SGP learns fundamental geometric principles rather than just memorizing specific data distributions.
Conclusion and Future Work
SGP proves that ground-truth labels are not a prerequisite for high-performance geometric perception. By leveraging robust estimation as a teacher, the framework reaches "Oracle" levels of accuracy. Future research may focus on increasing the convergence speed of the loop and exploring fully differentiable robust estimation layers to further integrate the teacher and student components.
Summary updated