Learning to Prune Feature Matches from Raw, Unlabeled Match Sets

When two photos of the same scene are matched by comparing local patch descriptors, most of the candidate matches are wrong. Up to 90% can be false according to LLHA-Net. Neural "correspondence pruners" can sort true matches from false ones far better than classical filters, but they learn this from labels. Each training pair needs a ground-truth camera pose, usually produced by running a full 3D reconstruction offline. This proposal asks whether a pruner can learn most of what it needs from the raw, contaminated match sets themselves, and then reach full-supervision quality with a small fraction of pose-labeled pairs.

The core idea is simple. A true match follows the same smooth motion as its neighbours, so a model that has seen the rest of the set can predict where it lands. A false match points somewhere arbitrary, so no amount of context predicts it. We train a weight-shared twin (Siamese) network, meaning two copies of one encoder with identical weights, on a masked prediction game over raw putative matches. The network must predict, from context alone, what a hidden match "looks like" to an encoder that sees everything. How badly each match fails this game becomes a label-free inlier score. Pretraining runs on contaminated sets, and fine-tuning then needs only a few labels.

Core Claim

Three prior lines of work almost meet this goal but each keeps a labeling dependency.

CorrMAE pretrains a correspondence transformer by masking and regressing match coordinates. It pretrains on verified inliers taken from MegaDepth's 3D reconstructions, so the outliers, which are the thing to be rejected, never appear during pretraining. ULMR avoids labels with reinforcement learning, rewarding the size of the consensus set behind a fitted fundamental matrix. It inherits the high variance of policy gradients and needs a fixed rigid model. The local-consensus Siamese network with reciprocal loss runs forward (A→B) and backward (B→A) match sets through shared weights and enforces symmetric predictions, but it is fully supervised by pose.

The obvious combination would make that forward/backward symmetry the self-supervised signal. It fails, because a network that outputs a constant satisfies it perfectly, so symmetry alone carries no information about which matches are true. Our claim is narrower and testable. The per-match error of a masked joint-embedding prediction game, played on raw putative sets, is an informative inlier signal by itself. Pretraining a pruner this way gives a better starting point than three alternatives: training from scratch, CorrMAE-style coordinate regression, and pretraining on RANSAC-derived pseudo-labels, with the gap largest at 1–10% labels.

Research Program

The input is a putative match set between images A and B. Each match $i$ carries a source coordinate $p_i$, a target coordinate $q_i$, and a descriptor ratio score. The game uses two encoders with the architecture of a standard permutation-invariant pruner (OANet's encoder, from OANet), so that any gain comes from pretraining and not from architecture.

The target encoder sees the full set. It is an exponential moving average of the anchor encoder, a slowly updated copy, as in Masked Siamese Networks (MSN).

The anchor encoder sees the same set with a random subset $M$ of matches masked. For each masked match it receives only the source coordinate $p_i$ as a query; the target coordinate $q_i$ is hidden. We borrow this coordinate-as-prompt trick from CorrMAE.

Both encoders map each match to a soft assignment over $K$ learnable prototypes. Prototypes are shared reference vectors; here we expect them to act as discrete "motion states." The anchor's distribution $a_i$ is trained to match the target's sharpened distribution $t_i$.

We use prototype assignment instead of coordinate regression for a specific reason. Under repetitive texture the plausible landing spots for a hidden match are multimodal, and a discrete distribution can represent several modes where a single regressed point cannot. To prevent collapse onto a single prototype we apply MSN's mean-entropy regularizer, which rewards using all prototypes evenly across the batch.

The per-match loss is

$$\ell_i = H(t_i, a_i), \qquad \mathcal{L} = \frac{\sum_{i \in M} w_i \,\ell_i}{\sum_{i\in M} w_i} - \lambda\, H(\bar a),$$

where $H(t_i,a_i)$ is the cross-entropy between target and anchor assignments, $\bar a$ is the batch-average anchor distribution, and $\lambda$ weights the anti-collapse entropy term.

The weight $w_i = \exp(-\ell_i/\tau)$, with $\tau$ a temperature and the weight computed without gradients, down-weights matches the model cannot predict. This is match-level outlier arbitration, a direct analogue of the view-level reweighting in UOTA. UOTA handled only mild contamination, so whether its logic survives 90% outliers is itself something we test.

At test time, the label-free inlier score for match $i$ is $s_i = -\ell_i$, averaged over several random masks.

Siamese weight sharing enters in one more place. Each set is also presented with A and B swapped, and both directions share prototypes. This makes the learned motion states direction-agnostic. It is an augmentation that rides on the prediction game, not the source of the signal.

Fine-tuning attaches the standard inlier-classification and essential-matrix losses to the pretrained encoder. We fine-tune on 1%, 5%, 10% and 100% of pose-labeled YFCC100M training pairs.

The analysis plan has three parts. First, we check whether masked-prediction error separates inliers from outliers before any labels are used. Second, we check whether prototypes do better than coordinate regression under repetitive indoor texture. Third, we check whether the match-level weighting is necessary or harmful at extreme outlier ratios, where self-paced weighting could lock in early mistakes.

Baselines

The naive reference is the same OANet-encoder pruner trained from random initialization on each label fraction. Beside it we run plain RANSAC, GMS (Grid-Based Motion Statistics) and LPM (Locality Preserving Matching), which need no labels at all and bound what "label-free" already buys. Every learned arm uses the same SIFT putative matches, the same backbone, the same pretraining images and epochs, the same fine-tuning schedule, and the same hyperparameter search budget on a held-out validation split.

The strongest full-supervision references come from the file.

BCLNet reports an F-score of 78.31% on YFCC100M Unknown. The local-consensus Siamese network reports an F-score of 72.16% on YFCC100M Unknown and mAP@5° of 19.78% on SUN3D Unknown. CorrMAE reports AUC@5° of 40.16% on YFCC100M pose estimation, against 34.51% for NCMNet. No reported result found on Lacuna for any correspondence pruner trained at reduced label fractions, so we generate all label-fraction curves ourselves.

Beating the field means one of two things. Either our 10%-label model matches our own 100%-label from-scratch run, or our curve dominates every pretraining baseline at 1% and 5%.

The strongest label-efficient competitor is CorrMAE's coordinate-regression pretraining, run in two forms. The first is its original form on MegaDepth reconstruction inliers; this gives it extra supervision we do not use and serves as an upper reference. The second is a fair form on RANSAC-filtered inliers from the same unlabeled images.

ULMR is reimplemented on the same backbone with its consensus reward. Its paper reports mean precision of about 93.8% under its own protocol, which is not directly comparable to ours.

Ablations isolate each ingredient: prototype assignment versus coordinate regression under identical masking, weighting on versus off, and direction swap on versus off. A further control replaces the masked-prediction game with the forward/backward symmetry loss alone, to confirm that symmetry is uninformative without masking.

Why not just…?

Why not just run RANSAC on the raw matches and train the pruner on its inlier masks as pseudo-labels? This is the most serious threat, and it may work well on outdoor scenes where RANSAC often succeeds. Its weakness is predictable. Where RANSAC fails, as with low texture and dominant planes in SUN3D, its pseudo-labels are confidently wrong, and the pruner learns those errors. Our score never commits to a single global model. We run this arm on the same backbone and budget. If it matches us on both YFCC100M and SUN3D, we will report that the masked game adds nothing over pseudo-labeling and recommend the simpler method. A middle outcome is also possible: the two signals help on different failure cases. We will then test feeding RANSAC masks as extra targets alongside our objective, rather than claim a win.

Why not just use CorrMAE pretraining and fine-tune it with few labels? CorrMAE needs verified inliers to pretrain, and verified inliers come from the same 3D reconstruction pipeline that produces pose labels. Its "self-supervision" therefore relies on the expensive supervision this proposal is trying to remove. The fair CorrMAE arm, which uses RANSAC inliers in place of reconstruction inliers, tests whether its advantage survives once that dependency is gone. The prototype-versus-regression ablation separately tests whether regression itself is the weak point under repetitive texture.

Why not just use ULMR's consensus reward? It optimizes the same intuition as ours, that true matches agree on one model, but through sampled subsets and policy gradients. Those gradients are noisy, and the method is tied to one rigid model per pair. Our objective gives every match a direct gradient. A matched-compute comparison of label-fraction curves and training stability settles which is better in practice.

Why not just train a supervised pruner on fewer labels? That is the from-scratch arm. If the from-scratch arm alone loses little F-score between 100% and 10% labels, the label bottleneck is smaller than claimed and the proposal's premise weakens. We report this curve first.

Proposed Experiments

Pretraining uses raw SIFT putative matches from YFCC100M training pairs and MegaDepth images. Poses and depth are discarded, so pretraining sees only image pairs and their candidate matches. Evaluation follows the standard Known/Unknown splits of YFCC100M and SUN3D. Metrics are precision, recall and F-score of match classification, and pose mAP/AUC at 5°, 10° and 20°. Transfer to Aachen Day-Night and HPatches follows CorrMAE's protocol.

The first decisive check needs no labels. We compute the area under the ROC curve of the inlier score $s_i$ against ground-truth epipolar inlier labels on YFCC100M Unknown and SUN3D Unknown, and compare it with the inlier masks from RANSAC, GMS and LPM. If masked-prediction error does not beat GMS and LPM here, the mechanism is not working and later gains would come from something else.

The second check is the label-fraction curves at 1%, 5%, 10% and 100% for every arm, with three seeds each. The third stratifies results by putative outlier ratio, binned up to the 90% regime, and by scene type. This is where the weighting ablation and the RANSAC pseudo-label comparison should diverge if our account is right: the advantage should concentrate in high-outlier and indoor bins where single-model consensus breaks down.