Abstract
Prior ReLoc3R achieves breakthrough performance with fast 25ms inference and state-of-the-art regression accuracy, yet our analysis reveals subtle geometric inconsistencies in its internal representations that prevent reaching the precision ceiling of correspondence-based methods like MASt3R (which require 300ms per pair). In this work, we present GeLoc3r, a novel approach to relative camera pose estimation that enhances pose regression methods through Geometric Consistency Regularization (GCR). GeLoc3r overcomes the speed-accuracy dilemma by training regression networks to produce geometrically consistent poses without inference-time geometric computation. During training, GeLoc3r leverages ground-truth depth to generate dense 3D-2D correspondences, weights them using a FusionTransformer that learns correspondence importance, and computes geometrically-consistent poses via weighted RANSAC. This creates a consistency loss that transfers geometric knowledge into the regression network. Unlike FAR method which requires both regression and geometric solving at inference, GeLoc3r only uses the enhanced regression head at test time, maintaining ReLoc3R's fast speed and approaching MASt3R's high accuracy. On challenging benchmarks, GeLoc3r consistently outperforms ReLoc3R, achieving significant improvements including 40.45% vs. 34.85% AUC@5° on the CO3Dv2 dataset (16% relative improvement), 68.66% vs. 66.70% AUC@5° on RealEstate10K, and 50.45% vs. 49.60% on MegaDepth1500. By teaching geometric consistency during training rather than enforcing it at inference, GeLoc3r represents a paradigm shift in how neural networks learn camera geometry, achieving both the speed of regression and the geometric understanding of correspondence methods.
Introduction
Estimating the relative position and orientation between two camera views is a fundamental task in computer vision, critical for applications such as Augmented Reality (AR), robotics, and autonomous navigation. Traditionally, this field has been divided by a "speed-accuracy dilemma." On one hand, correspondence-based methods (e.g., MASt3R) achieve high precision by matching dense point sets and solving geometric equations, but their high computational cost (~300ms per pair) limits real-time use. On the other hand, regression-based methods (e.g., ReLoc3R) offer near-instantaneous inference (~25ms) by predicting poses directly from images, yet they often struggle to reach the accuracy of geometric solvers because they rely on semantic patterns rather than explicit 3D geometry.
GeLoc3r addresses this gap by introducing Geometric Consistency Regularization (GCR), a framework that allows regression models to inherit the geometric precision of correspondence methods without sacrificing inference speed. By identifying and correcting "descriptor inconsistency"—where predicted poses fail to align visual features correctly—GeLoc3r pushes the boundaries of what one-shot regression models can achieve.
Methodology
The core of the GeLoc3r framework is the Geometric Consistency Regularization (GCR) training strategy. This approach uses "privileged" information, such as ground-truth depth, during the training phase to enforce geometric constraints. Crucially, this extra information is not required during inference, allowing the model to remain fast, using only a regression-only architecture at test time.
Key Components
- FusionTransformer: This module processes pairs of image descriptors to learn the relative importance of different scene regions. It identifies stable geometric features while down-weighting outliers like dynamic objects or repetitive textures.
- Weighted RANSAC Solver: During training, the model uses the weights from the FusionTransformer alongside ground-truth depth to calculate a geometrically optimal pose. This "solved" pose serves as a high-quality target for the regression head.
- Dual Supervision: The training objective combines a standard pose loss with a consistency loss. This ensures that the regression head matches the geometric solver's output and that 3D points projected between views align with their corresponding visual descriptors.
The system is built upon a Vision Transformer (ViT) backbone, leveraging pre-trained weights from the MASt3R model to provide a strong foundation for feature extraction.
Experiments and Results
GeLoc3r was trained on a massive collection of 10 diverse datasets, including ScanNet++, MegaDepth, and RealEstate10K, and evaluated on standard benchmarks to test both accuracy and generalization.
As shown in the figure above, GeLoc3r effectively reduces the "Cosine Similarity Error," leading to much tighter alignment between views. Quantitatively, the model demonstrates superior performance across multiple metrics:
- CO3Dv2: Achieved 40.45% AUC@5°, a 16% relative improvement over ReLoc3R.
- MegaDepth1500: Reached 50.45% AUC@5°, showcasing robustness in outdoor environments.
- Visual Localization: On the Cambridge Landmarks dataset, GeLoc3r reduced the maximum trajectory error by 58% compared to the baseline.
- Inference Speed: The model maintains a 33ms inference time, which is over 60x faster than traditional correspondence-based solvers.
Conclusion
GeLoc3r demonstrates that geometric constraints can be effectively "baked into" a neural network's weights during training, eliminating the need for expensive geometric computation at runtime. By bridging the gap between regression and correspondence methods, GeLoc3r provides a scalable and efficient solution for real-time 3D vision tasks. Future research will focus on removing the dependency on dense ground-truth depth during training and extending the framework to multi-view consistency.
Summary updated