Normalized Matching Transformer

Abtin Pourhadi, Paul Swoboda

Introduction

Keypoint matching—identifying the same physical point on an object across different images—is a foundational challenge in computer vision. Traditionally, this task was approached as a hybrid problem: neural networks extracted features, while separate, often non-differentiable combinatorial solvers handled the matching. These pipelines are notoriously complex and difficult to train end-to-end. The Normalized Matching Transformer (NMT) addresses these issues by introducing a streamlined, "pure" deep learning pipeline that outperforms complex traditional systems.

Proposed Architecture

The NMT architecture consists of five distinct building blocks designed to exploit the geometric and statistical properties of the matching task:

  1. Visual Feature Extraction: Utilizing a Swin-Transformer backbone, the model extracts rich, multi-scale features, providing a significant upgrade over older VGG-based standards.
  2. Geometric Refinement (SplineCNN): A SplineCNN Graph Neural Network adds an inductive bias, helping the model understand the relative spatial layout and geometry of keypoints.
  3. Normalized Transformer (nGPT): This core component uses hyperspherical normalization throughout the decoder. By keeping features on a unit sphere at every step, the architecture ensures stable gradients and faster convergence.
  4. Sinkhorn Matching: To ensure valid one-to-one matching, the model employs a differentiable Sinkhorn algorithm to approximate linear assignment.
  5. Advanced Losses: Training is guided by a combination of InfoNCE (contrastive loss) and a Hyperspherical loss, ensuring features are spread uniformly across the feature space to maximize distinctiveness.
The NMT architecture pipeline, featuring a Swin-Transformer backbone, SplineCNN refinement, and a Normalized Decoder Transformer.

Key Insights: The Power of Normalization

A central argument of the NMT is that because matching ultimately relies on cosine similarity—which occurs on a hypersphere—the network should maintain its internal processing on that sphere. This continuous normalization allows for significantly faster training and higher final accuracy. Furthermore, the results suggest that the field has been limited by a reliance on aging backbones; upgrading to modern architectures like the Swin-Transformer yields substantial performance gains.

Detail of the loss computation involving InfoNCE and Hyperspherical losses applied to the normalized features.

Experimental Results

The model was evaluated on two major benchmarks: PascalVOC and the more challenging SPair-71k.

  • State-of-the-Art Accuracy: NMT outperforms previous models (such as BBGM and GMTR) by 5.1% on PascalVOC (88.7%) and 2.2% on SPair-71k (86.7%).
  • Training Efficiency: The model achieves these results in just 6 epochs, representing a 1.7x to 2.6x speedup over competitors that typically require 10 to 16 epochs.
  • Real-Time Viability: With an inference speed of approximately 44.4ms per image pair on an RTX 4090, the system is suitable for real-time applications.

Conclusion

By focusing on high-quality feature representations and consistent normalization, the Normalized Matching Transformer demonstrates that pure machine learning approaches can surpass complex combinatorial solvers. While the current implementation faces some challenges with kernel fusion optimization and extreme occlusions, it sets a new standard for efficiency and accuracy in sparse keypoint matching.

Summary updated

Abstract

We introduce the Normalized Matching Transformer (NMT), a deep learning approach for efficient and accurate sparse keypoint matching between image pairs. NMT consists of a strong visual backbone, geometric feature refinement via SplineCNN, followed by a normalized transformer for computing matching features. Central to NMT is our hyperspherical normalization strategy: we enforce unitnorm embeddings at every transformer layer and train with a combined contrastive InfoNCE and hyperspherical uniformity loss to yield more discriminative keypoint representations. This novel architecture/loss combination encourages close alignment of matching image features and large distance between non-matching ones not only at the output level, but for each layer. Despite its architectural simplicity, NMT sets a new state-of-the-art performance on PascalVOC and SPair-71k, outperforming BBGM (Rol´ınek et al. 2020), ASAR (Ren et al. 2022), COMMON (Lin et al. 2023) and GMTR (Guo et al. 2024) by 5.1% and 2.2%, respectively, while converging in at least ≥ 1.7× fewer epochs compared to other state of the art baselines. These results underscore the power of combining pervasive normalization with hyperspherical learning for geometric matching tasks.
BibTeX
@misc{pourhadi2025normalized,
  author = {Abtin Pourhadi and Paul Swoboda},
  title = {{Normalized Matching Transformer}},
  year = {2025},
  eprint = {2503.17715},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.17715},
}
Figure