Normalized Matching Transformer

Abtin Pourhadi, Paul SwobodaSubmission to ICLR 2026 · Withdrawn

TL;DR

The Normalized Matching Transformer (NMT) is a streamlined, pure deep-learning pipeline for sparse keypoint matching that eliminates the need for complex combinatorial graph-matching solvers. By strictly constraining feature vectors to the unit hypersphere throughout the transformer decoder and combining contrastive InfoNCE with progressive hyperspherical uniformity losses, NMT achieves state-of-the-art accuracy on PascalVOC and SPair-71k while converging in at least $1.7\times$ fewer training epochs.

What Problem Does This Paper Address?

Sparse keypoint matching aims to establish semantically meaningful one-to-one correspondences between distinctive landmark points across different instances of an object class (e.g., matching a dog’s ear or a plane’s wing across distinct images and environments).

Historically, this problem has been framed as a Quadratic Assignment Problem (QAP) or graph matching, requiring NP-hard combinatorial solvers. These solvers are computationally expensive, typically run on CPUs, and are difficult to integrate smoothly into end-to-end backpropagation pipelines. While recent neural approaches have started replacing combinatorial routines with transformer decoders and Sinkhorn-based approximations, their feature representations often lack discriminative power and train slowly. NMT sets out to prove that keypoint matching performance has not yet saturated, and that pervasive architectural normalization paired with hyperspherical representation learning can outperform complex hybrid solvers.

Methods

NMT formulates keypoint matching as an end-to-end learning pipeline consisting of five components:

  1. Visual Feature Extraction: A pre-trained Swin-Large visual backbone extracts multi-scale features, bilinear-interpolated at keypoint coordinates and concatenated. A global mean-pooled image token provides category conditioning.
  2. Geometric Feature Refinement (SplineCNN): A 2-layer SplineCNN graph neural network operates over a Delaunay triangulation of keypoints to aggregate local spatial layout and Euclidean geometry into the node features.
  3. Normalized Transformer Decoder: Rather than using a standard transformer, NMT uses a 4-layer normalized transformer (nGPT) architecture. Feature representations are projected back onto the unit hypersphere (Norm(·)) immediately after self-attention, cross-attention, MLP blocks, and residual updates. Features are also modulated with the global feature token.
  4. Hyperspherical & Contrastive Losses:
  5. InfoNCE Loss: Applied symmetrically across images to pull matching cross-image keypoint embeddings together while pushing non-matching pairs apart.
  6. Hyperspherical Uniformity Loss: Penalizes the maximum cosine similarity between distinct keypoints within the same image, spreading them uniformly across the hypersphere. This loss is also injected as an auxiliary layer-wise loss, weighted progressively deeper in the network ($k \cdot p$, where $p=0.3$).
  7. Inference via Sinkhorn: At test time, raw pairwise cosine similarities pass through a differentiable log-space Sinkhorn algorithm, followed by a greedy row-wise maximum extraction. No combinatorial solvers are used.

Datasets

The authors evaluate on two primary sparse keypoint matching benchmarks:

  • PascalVOC: Evaluated with Berkeley annotations across 20 object classes (images sized $256 \times 256$, up to 23 keypoints per image). Standard intersection filtering is used (evaluating only keypoints present in both images).
  • SPair-71k: A larger, higher-quality successor to PascalVOC containing 70,958 image pairs across 18 categories with richer viewpoint and scale variations.

No new datasets were introduced; simpler benchmarks like Willow Object Class were omitted as existing methods are already near-perfect on them.

Key Insights and Ideas

  • Hypersphere-Centric Consistency: Keypoint affinity fundamentally relies on cosine similarity. Constraining intermediate and residual transformer representations to the unit hypersphere at every stage—rather than just normalizing once before loss calculation—stabilizes training dynamics, prevents internal feature magnitude drift, and substantially accelerates convergence.
  • Deep Feature Repulsion: Spreading keypoint features uniformly over the hypersphere inside each individual image prevents feature collapse. Weighting this hyperspherical loss progressively higher in deeper transformer layers ensures final matching features are maximally discriminative.
  • Simpler Beats Hybrid: Highly structured combinatorial solvers are no longer necessary. A well-regularized transformer decoder with simple Sinkhorn decoding matches or surpasses state-of-the-art hybrid solvers.

Key Results

  • State-of-the-Art Accuracy:
  • PascalVOC: Reaches 88.7% accuracy, outperforming leading baselines like GMTR (83.6%), COMMON (82.7%), and BBGM (80.1%) by +5.1%, leading in 17 out of 20 categories.
  • SPair-71k: Reaches 86.7% accuracy, outperforming COMMON (84.5%) and GMTR (83.2%) by +2.2%, leading in 13 out of 18 categories.
  • Faster Convergence: Reaches convergence in just 6 epochs (compared to 10 for BBGM and 16 for ASAR and COMMON), representing a $\ge 1.7\times$ reduction in training epochs.
  • Inference Runtime: Takes 44.4 ms per image pair on a single RTX 4090 (39 ms for Swin-Large + SplineCNN, and only 5.4 ms for the two transformer decoders).
  • Ablation Findings: The combined InfoNCE/hyperspherical loss is the primary driver (+15.1% over standard cross-entropy). Replacing the normalized transformer with a vanilla transformer drops accuracy by 2.6%, and replacing Swin with VGG16 causes a 4.9% drop (though NMT with VGG16 still scores 83.8%, beating prior VGG-based baselines).

Limitations and Future Work

  • Kernel Implementation: While the model requires fewer epochs to converge, normalized transformers currently lack optimized fused GPU kernels, resulting in slightly longer wall-clock time per epoch than standard vanilla transformers.
  • Augmentation Sensitivity: Standard spatial augmentations (Mixup, CutMix, Random Erasing) improved performance (+1.2%), but pixel-level photometric augmentations (RGB shifts, random gamma, saturation shifts) degraded matching performance.
  • Future Directions: Extending pervasive hyperspherical normalization and uniformity losses to dense correspondence, homography estimation, and other structured geometric learning tasks.

Summary updated

Abstract

We introduce the Normalized Matching Transformer (NMT), a deep learning approach for efficient and accurate sparse keypoint matching between image pairs. NMT consists of a strong visual backbone, geometric feature refinement via SplineCNN, followed by a normalized transformer for computing matching features. Central to NMT is our hyperspherical normalization strategy: we enforce unitnorm embeddings at every transformer layer and train with a combined contrastive InfoNCE and hyperspherical uniformity loss to yield more discriminative keypoint representations. This novel architecture/loss combination encourages close alignment of matching image features and large distance between non-matching ones not only at the output level, but for each layer. Despite its architectural simplicity, NMT sets a new state-of-the-art performance on PascalVOC and SPair-71k, outperforming BBGM (Rol´ınek et al. 2020), ASAR (Ren et al. 2022), COMMON (Lin et al. 2023) and GMTR (Guo et al. 2024) by 5.1% and 2.2%, respectively, while converging in at least ≥ 1.7× fewer epochs compared to other state of the art baselines. These results underscore the power of combining pervasive normalization with hyperspherical learning for geometric matching tasks.
BibTeX
@misc{pourhadi2025normalized,
  author = {Abtin Pourhadi and Paul Swoboda},
  title = {{Normalized Matching Transformer}},
  year = {2025},
  eprint = {2503.17715},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.17715},
}
Figure