Fast neural feature matchers find the same point in two photos by passing keypoints through a stack of attention layers. They save time by stopping early once each point is confident about its partner. That rule asks the wrong question. The downstream goal is a camera pose, and the pose is often settled long before every keypoint is confident. Current fast matchers also cannot notice when the pose is settled, because they output bare $(x, y)$ matches. A single match carries too little geometry to test a pose hypothesis cheaply.
We propose a matcher that, at every layer, also predicts an affine correspondence for each tentative match. An affine correspondence is a small $2\times 2$ matrix describing how the image patch around a point stretches, rotates and shears between the two views. The matcher starts from the scale and orientation the keypoint detector already provides and corrects this estimate a little at each layer. Because two affine correspondences are enough to fix the relative pose of calibrated cameras, the network can run a tiny pose search inside itself after each layer. It stops when that pose stops changing. The same affine outputs then feed a two-sample solver in the final robust estimation stage, which cuts that stage's cost as well. In two sentences: make the matcher output patch deformations, not just points, and use them to decide when to stop computing based on whether the camera pose has converged rather than whether individual points are confident.
Core Claim
We claim that geometric consensus built from refined affine correspondences is a better stopping signal than per-point confidence. At matched end-to-end wall-clock time, covering both matching and pose estimation, the resulting matcher should reach higher pose accuracy than a confidence-exit point matcher followed by point-based RANSAC, the standard loop that samples random subsets of matches and keeps the pose most matches agree with. It should also approach the accuracy of heavy two-stage affine matchers at a fraction of their cost.
The saving has a simple source. RANSAC needs about
$$N = \frac{\log(1-p)}{\log(1-w^{s})}$$
samples to find an all-inlier sample with probability $p$, where $w$ is the inlier fraction and $s$ is the minimal sample size. Dropping $s$ from 5 points to 2 affine correspondences makes an in-network pose check cheap enough to run after every layer. At $w=0.3$ and $p=0.99$, the formula gives about 49 samples instead of about 1,900.
Relation to Prior Work
Four lines of work bear on the design, each in a different role.
The first supplies the motivation. LightGlue showed that adaptive depth (exit classifiers at each layer) and adaptive width (pruning unmatchable points) make attention matchers 2–4x faster than SuperGlue. However, it treats keypoints as zero-order positions and discards detector scale and orientation. Its exit criterion is local confidence. We keep its architecture and replace both the output type and the stopping rule.
The second supplies the geometry. Raposo and Barreto derived the two-affine-correspondence essential-matrix solver we run inside the network. Relative Pose from SIFT Features showed that the scale and orientation SIFT computes anyway constrain epipolar geometry, the geometric relation between two camera views. We use this to seed the affine estimate at no cost. Barath and Hajder showed that one affine correspondence suffices under planar vehicle motion, which we test on KITTI.
The third is our closest competitor. Learning Affine Correspondences by Integrating Geometric Constraints learns affine shape with an "Affine Sampson Distance" loss grounded in the fundamental matrix. It reaches the best MegaDepth accuracy in the file, but it freezes dense point matches first and then crops patches in a second pass. Refinement there happens only through the training loss, never iteratively at test time. We reuse its loss but fold affine estimation into the matcher's layers, where it can be refined, pruned and used for exit.
ARS-MAGSAC likewise feeds scale and orientation into a network trained with an affine epipolar error. Its network scores matches once, and the only iteration is RANSAC reordering. The feature attributes themselves are never corrected.
The fourth constrains the design. Iterative Deep Homography Estimation found that weight-tied iterative refinement beats stacking layers and is about 8x faster than classical Lucas-Kanade iteration. That supports learned residual updates. Its explicit 4D correlation volume is memory-heavy, however, so we refine sparse per-match affine parameters rather than a dense field. Semicalibrated Relative Pose from an Affine Correspondence and Monodepth warns that affine-based solvers degenerate on fronto-parallel planes. Our exit rule must therefore fall back to the full layer budget when the tiny pose search is ill-conditioned, not exit on a spurious consensus.
Research Program
We would build on a LightGlue-style matcher operating on SIFT and on SuperPoint keypoints.
Each match token carries a four-parameter affine state. It is initialized from detector scale and orientation for SIFT, and from the identity for SuperPoint, which serves as a no-metadata control. A small head after each layer predicts a residual update to this state from the self- and cross-attended descriptors of both endpoints. Training combines LightGlue's matching loss with the Affine Sampson Distance at every layer. On synthetic homography pairs we add direct supervision from the exact local affine map, which is the Jacobian of the homography at the point.
After each layer, the exit module takes the top-$K$ confident tentative matches. It runs a fixed small number of two-affine-correspondence hypotheses on the GPU and scores each hypothesis by affine Sampson distance over all tentative matches. It exits when two consecutive layers agree: their best hypotheses differ by less than a rotation threshold, and their inlier sets overlap above a threshold. A conditioning check on the sampled correspondences blocks exit in near-degenerate configurations.
Points already pruned by adaptive width keep their last affine state. This directly tests whether pruning tokens is compatible with regressing continuous deformations.
A secondary variant feeds the current best epipolar hypothesis back into the next layer as an extra positional cue, closing the feedback loop that prior affine solvers lack. We treat this variant as exploratory.
Baselines
The naive reference is mutual nearest-neighbour matching of SIFT descriptors followed by five-point RANSAC. It shows how much any learned matcher adds and gives the zero-cost latency floor.
The strongest latency competitor is LightGlue with its own confidence-based exit and pruning, followed by five-point RANSAC. This is exactly the design we claim to improve on. We would retrain it on the same MegaDepth split, with the same keypoints, number of layers, GPU and RANSAC implementation, and give it the same hyperparameter-search budget as our model. We then sweep its exit threshold to trace an accuracy-versus-time curve. Beating it means our curve lies above its curve, measured as AUC@5° against total milliseconds per pair, across the operating range, not only at one point. The file contains no MegaDepth AUC for LightGlue ("no reported result found on Lacuna for LightGlue pose AUC"), so we would report our own reproduction.
The strongest accuracy competitor is the two-stage learned affine matcher of Learning Affine Correspondences by Integrating Geometric Constraints with GC-RANSAC. It reports an AUC of 65.5 at 5° on MegaDepth, against 62.6 for RoMa and 60.4 for DKM. We do not expect to beat it outright with a sparse matcher. Success means coming within a few AUC points at several times lower latency, timed by us on identical hardware.
On PhotoTourism, the reference point is ARS-MAGSAC. It reports AUC@10° of 0.50 at 33.9 ms per pair. SuperGlue reports AUC@20° of 64.2%.
Three ablations isolate our key ingredients. The first replaces the learned affine refinement with raw detector frames, passed unchanged through the matcher. The second removes the geometric exit and uses LightGlue's confidence exit on the affine-refining model. The third keeps the geometric exit but uses the final-layer affine estimate only, with no per-layer refinement.
Why not just…?
Why not just run LightGlue and five-point RANSAC, since RANSAC is already fast on good matches? This is the most serious threat. When the inlier fraction is high, RANSAC time is small and the minimal-sample advantage shrinks. Any gain must then come from earlier exit inside the matcher. We therefore report total time, not solver time. We also stratify pairs by inlier fraction and overlap. If our method helps only on the hard, low-overlap strata, we will say so. If it never helps at matched time, the core claim fails.
Why not just feed the SIFT scale and orientation the detector already gives into the SIFT-feature solver of Kukelova and Barath? That costs nothing extra and may capture most of the benefit. The raw-frame ablation tests exactly this. We would measure affine error against homography ground truth on HPatches, and pose AUC on MegaDepth, with and without learned refinement. If refinement does not reduce affine error across layers or improve pose, the extra machinery is unjustified. The model should then be reduced to a better-seeded exit rule.
Why not just refine affine frames after matching with classical per-patch Lucas-Kanade? IHN reports that classical inverse-compositional iterators are about 8x slower than learned ones and suffer rank-deficient updates. FeatureBooster shows that learned features survive day-night appearance change far better than raw pixels (ORB night localization rises from 31.9% to 49.2%). Batched per-patch refinement on a GPU might still be cheap for two thousand matches, though. We would run it as a post-hoc baseline on LightGlue matches and time it honestly. Classical refinement also cannot inform the exit decision, which is the part of our proposal it cannot replicate.
Why not just use a heavy single-pass dense matcher such as RoMa? It is more accurate on hard pairs (88.0 mAA on IMC2022). However, it spends the same compute on easy and hard pairs. We place it on the accuracy-time plot as an upper-right anchor rather than a competitor in the low-latency regime.
Proposed Experiments
We train on MegaDepth and evaluate relative pose there and on PhotoTourism, using AUC at 5°, 10° and 20° against total per-pair time on one GPU. On ScanNet we test indoor, low-texture, near-planar scenes, where the degeneracy fallback matters.
On HPatches we measure per-layer affine error against homography Jacobians. This directly shows whether the per-layer refinement converges and whether pruned points keep usable affine estimates. On KITTI we additionally swap in the one-affine-correspondence planar-motion solver to test the extreme small-sample case.
Four checks are decisive. First, at matched total time, geometric exit must beat confidence exit on MegaDepth AUC@5°. Second, learned refinement must beat raw detector frames and post-hoc Lucas-Kanade on both affine error and pose. Third, the average exit layer must track scene difficulty, meaning it exits earlier on high-overlap pairs, without accuracy collapsing on fronto-parallel ScanNet scenes. Fourth, the gap to the two-stage affine matcher's 65.5 AUC@5° must be small relative to the latency saved. If the first check fails, the proposal reduces to an affine-output matcher, which is useful but modest. If the second fails, the right conclusion is that detector metadata should be used as-is.
References
- LightGlue: Local Feature Matching at Light Speed
- SuperGlue: Learning Feature Matching With Graph Neural Networks
- Theory and Practice of Structure-From-Motion Using Affine Correspondences
- Relative Pose from SIFT Features
- Relative planar motion for vehicle-mounted cameras from a single affine correspondence
- Learning Affine Correspondences by Integrating Geometric Constraints
- Adaptive Reordering Sampler with Neurally Guided MAGSAC
- Iterative Deep Homography Estimation
- Semicalibrated Relative Pose from an Affine Correspondence and Monodepth
- FeatureBooster: Boosting Feature Descriptors with a Lightweight Neural Network
- RoMa: Robust Dense Feature Matching