Manifold Embedding of Deep Image Features for Image Matching via Neural Adjoint Maps

Nafie El Amrani, Moritz Heep, Florian BernardarXiv 2026 · arXiv

TL;DR

This paper introduces a novel zero-shot framework for semantic image matching by treating 2D images as manifolds embedded in high-dimensional feature space. By adapting 3D shape-matching techniques—specifically functional maps and Neural Adjoint Maps—the authors achieve state-of-the-art dense correspondence results that are both smoother and significantly faster than existing spectral methods.

What Problem Does This Paper Address?

Finding "semantic correspondences"—matching the same part of an object across different images (e.g., a cat's ear to another cat's ear)—is a cornerstone of computer vision. While modern "foundation models" like DINOv2 provide incredibly rich feature descriptors, simply using a nearest-neighbor (NN) search to match them often leads to noisy, spatially inconsistent results.

The challenge is regularization: how do we ensure that the matching respects the global structure of the object? In the 3D world, researchers use Functional Maps, which operate on the intrinsic geometry of a shape. However, 2D images are "flat" grids, lacking the non-trivial geometry needed for these powerful spectral methods to work effectively. Previous attempts to bridge this gap relied on ad-hoc graph constructions that lacked a solid geometric foundation.

Methods

The authors propose a pipeline that treats an image not as a flat grid, but as a "wrinkled" surface embedded in the high-dimensional space of deep features.

  1. Manifold Embedding: They interpret high-dimensional features (from DINOv2 or Stable Diffusion) as vertex coordinates for the image grid. This transforms the flat 2D domain into a 2D manifold in $R^d$.
  2. Discrete LBO via Galerkin's Method: To define the geometry of this manifold, they derive a discrete Laplace-Beltrami Operator (LBO). Unlike previous work that used simple pixel-distance graphs, they use finite element theory (Galerkin’s method) to calculate mass and stiffness matrices based on how the deep features distort the surface.
  3. Neural Adjoint Maps (NAM): Standard functional maps are linear, which can lose information when the spectral basis is truncated (cut off at a low number of dimensions). The authors instead use a Neural Adjoint Map, a lightweight MLP that aligns the source and target spectral embeddings non-linearly.
The pipeline extracts features, builds a manifold-based LBO, and optimizes a NAM to find correspondences.

Datasets

The method was evaluated across several diverse benchmarks:

  • TSS (Taniai Benchmark): 400 image pairs for dense correspondence (cars, dogs, etc.).
  • SPair-71k: A large-scale benchmark for sparse keypoint matching across 18 categories.
  • FAUST-2D: A newly introduced synthetic benchmark derived from 3D human scans, providing ground-truth dense correspondences under various rotations.
  • NIH Chest X-ray: Used as a proof-of-concept for medical image registration.

Key Insights and Ideas

  • Geometry from Features: The "Aha!" moment is realizing that deep features are the geometry. By treating feature vectors as coordinates, the LBO naturally captures semantic boundaries. For example, the eigenfunctions of their LBO are much smoother and respect object silhouettes better than standard graph-based methods.
  • Spectral Regularization: By matching in the spectral domain (using the first $k$ eigenfunctions), the method inherently regularizes the flow. It filters out high-frequency noise that plagues raw nearest-neighbor matching.
  • Efficiency: Because the NAM is a lightweight MLP optimized per-image-pair, the method is incredibly fast. It achieves a 100x speedup over the previous state-of-the-art spectral method (Cheng et al.), dropping runtime from ~95 seconds to under 1 second per pair.
Qualitative results show that the proposed method produces much smoother and more spatially coherent mappings compared to baselines.

Key Results

  • State-of-the-Art Dense Matching: On the TSS dataset, the method achieved a [email protected] of 86.7, outperforming all other zero-shot baselines (the next best was CleanDIFT at 85.5).
  • Robust Sparse Matching: On SPair-71k, it showed superior robustness in distinguishing left vs. right keypoints and handling substantial pose variations.
  • Speed: The optimization takes only 0.76 seconds on a single NVIDIA L40 GPU, making it practical for real-world applications.
  • Ablation Success: Figure 5 demonstrates that the combination of their specific LBO and the NAM consistently outperforms using a standard graph Laplacian or linear functional maps.
Figure 1: We propose a novel zero-shot method for finding correspondences between different object instances from similar categories. In each panel, the left image is the source and the right is the target. Our method is flexible and solves (a) dense image matching, (b) sparse image matching, and due to the zero-shot nature it is directly applicable to other domains, such as (c) medical images.

Limitations and Future Work

  • Full Correspondence Assumption: The current formulation assumes every point in the source image has a counterpart in the target. It struggles with heavy occlusion or parts that appear in one image but not the other.
  • Semantic Symmetries: Like many spectral methods, it can occasionally "flip" a match (e.g., matching a left wing to a right wing) if the feature embedding is near-symmetric.
  • Feature Resolution: The precision is currently limited by the resolution of the underlying foundation model patches (e.g., DINO's 14x14 patches). Extending this to pixel-level extractors is a suggested next step.
  • Partial Matching: The authors note that adapting techniques from 3D partial shape matching could help address the occlusion issue in the future.

Summary updated

Abstract

Estimating semantic correspondences between different object instances of similar categories in real-world images is a fundamental challenge in computer vision. While the recent progress in foundation models has significantly advanced solutions for such difficult image correspondence problems, respective methods are often insufficiently regularised. For example, common nearest-neighbour matching with foundation model features often results in noisy correspondences, which is particularly prominent when solving for dense correspondences. In this work, we tackle image matching via functional maps, which have been popularised for 3D shape matching due to their powerful spectral formalism utilising an efficient low-dimensional linear representation. However, functional maps require that domains (in our cases the rectangular image grids) are equipped with an informative and non-trivial geometry. To define a meaningful manifold structure for images, we use Galerkin's method from finite elements to derive a discrete Laplace-Beltrami operator (LBO) based on a manifold embedding of deep image features. Further, we leverage a neural adjoint map that allows to represent non-linear mappings. We demonstrate that our method sets the new state of the art among zero-shot image correspondence methods on multiple dense correspondence benchmarks of real-world images. Our code is publicly available: https://github.com/NafieAmrani/NAMI
BibTeX
@misc{amrani2026manifold,
  author = {Nafie El Amrani and Moritz Heep and Florian Bernard},
  title = {{Manifold Embedding of Deep Image Features for Image Matching via Neural Adjoint Maps}},
  year = {2026},
  eprint = {2611.05083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05083},
}
Figure