Gromov Wasserstein Optimal Transport for Semantic Correspondences

Francis Snelgar, Stephen Jay Gould, Ming Xu, Liang Zheng, Akshay Asthana

Abstract: Establishing correspondences between image pairs is a long-studied problem in computer vision. With recent large-scale foundation models showing strong zero-shot performance on downstream tasks, there has been interest in using their internal feature maps for semantic correspondence. Recent works observe that features from DINOv2 and Stable Diffusion (SD) are complementary: DINOv2 produces accurate but sparse correspondences, while SD produces spatially consistent ones. Consequently, current state-of-the-art methods ensemble both, which is computationally expensive. In this work, we replace SD features with a superior matching algorithm imbued with spatial consistency. Specifically, we replace standard nearest neighbors matching with an optimal transport algorithm including a Gromov-Wasserstein spatial smoothness prior. We significantly boost DINOv2 performance, rivaling or surpassing state-of-the-art SD-based methods while being 5–10x more efficient. Code: https://github.com/fsnelgar/semantic_matching_gwot.

Introduction: The Computational Bottleneck of Semantic Matching

Semantic correspondence involves finding matching parts between different instances of the same object class (e.g., mapping the ear of a golden retriever to the ear of a pug). This task is notoriously difficult due to variations in pose, scale, and non-rigid deformations.

Recently, researchers have achieved impressive results by extracting internal feature maps from large foundation models. DINOv2 provides highly accurate but spatially sparse matches, while Stable Diffusion (SD) provides excellent spatial consistency. To get the best of both worlds, current state-of-the-art (SOTA) approaches ensemble features from both models. However, running multiple billion-parameter models just for feature extraction is slow and memory-intensive. This paper asks: can we achieve the spatial consistency of Stable Diffusion without actually using it, relying solely on the lightweight DINOv2 model?

Methods: Upgrading the Matching Algorithm

Instead of upgrading the feature extractor, the authors upgrade the matching algorithm. Standard methods use simple Nearest Neighbor (NN) matching. This paper replaces NN with an Optimal Transport (OT) framework that explicitly encodes the desirable properties of semantic matching:

  1. Feature Similarity: Uses standard cosine similarity of DINOv2 features as the base cost.
  2. Spatial Smoothness (Gromov-Wasserstein): This is the core novelty. They use a Gromov-Wasserstein (GW) OT formulation, which penalizes matches where neighboring pixels in the source image map to distant pixels in the target image. To avoid the massive computational cost of standard GW matrix multiplications, they implement this using highly optimized 2D convolutions.
  3. Object Symmetry: Local features can be ambiguous (e.g., left vs. right wheels). They introduce a symmetry-aware objective that enforces horizontal ordering to respect the global pose.
  4. Unbalanced Formulation: Standard OT forces every pixel to match. Because of occlusions, they relax this constraint using a Kullback-Leibler (KL) divergence penalty, allowing the model to leave some pixels unmatched.
Visualization of the spatially consistent correspondences achieved by the GWOT algorithm.

Key Results: Efficiency and Accuracy

The authors evaluate their method on three standard benchmarks: TSS (rigid objects), PF-PASCAL (pose/scale variation), and SPair-71k (large-scale, extreme variations). The results demonstrate that mathematical priors can effectively replace billions of parameters.

Performance vs. Efficiency across TSS, PASCAL, and SPair-71k benchmarks. The GWOT method (Ours) occupies the top-left quadrant, indicating high accuracy and low latency.
  • Massive Efficiency Gains: The proposed method requires only 217ms per image pair and relies on an 87M parameter model. In contrast, SD-based SOTA methods (like DIFT or ToTF) take between 1,900ms and 2,300ms and require >1.2B parameters—an 8-10x speedup.
  • State-of-the-Art Accuracy: On the TSS dataset, the method achieves a new SOTA (92.3% average PCK vs. the previous 88.7%).
  • Highly Competitive: On the challenging SPair-71k dataset, the GWOT method boosts the DINOv2 baseline from 55.7% to 62.3%, closing the gap with heavy ensemble methods at a fraction of the cost.

Discussion and Limitations

The primary insight is that spatial consistency is a geometric property that can be enforced during matching rather than being "learned" by a generative model. However, limitations remain:

  • Extreme Scale Changes: Fixed-radius spatial penalties can struggle when scale differences are extreme.
  • Outlier Smoothing: If initial matches are overwhelmingly wrong, the GW prior can reinforce incorrect "smooth" regions.
  • Symmetry Assumptions: The symmetry loss assumes moderate pose variations and may fail in extreme opposite-viewpoint comparisons.

Future work will explore adaptive filter sizes and extending this efficient framework to video label propagation tasks.

Summary updated

Abstract

Establishing correspondences between image pairs is a long studied problem in computer vision. With recent large-scale foundation models showing strong zero-shot performance on downstream tasks including classification and segmentation, there has been interest in using the internal feature maps of these models for the semantic correspondence task. Recent works observe that features from DINOv2 and Stable Diffusion (SD) are complementary, the former producing accurate but sparse correspondences, while the latter produces spatially consistent correspondences. As a result, current state-of-the-art methods for semantic correspondence involve combining features from both models in an ensemble. While the performance of these methods is impressive, they are computationally expensive, requiring evaluating feature maps from large-scale foundation models. In this work we take a different approach, instead replacing SD features with a superior matching algorithm which is imbued with the desirable spatial consistency property. Specifically, we replace the standard nearest neighbours matching with an optimal transport algorithm that includes a Gromov Wasserstein spatial smoothness prior. We show that we can significantly boost the performance of the DINOv2 baseline, and be competitive and sometimes surpassing state-of-the-art methods using Stable Diffusion features, while being 5--10x more efficient. We make code available at https://github.com/fsnelgar/semantic_matching_gwot .
BibTeX
@inproceedings{snelgar2025gromov,
  author = {Francis Snelgar and Stephen Jay Gould and Ming Xu and Liang Zheng and Akshay Asthana},
  title = {{Gromov Wasserstein Optimal Transport for Semantic Correspondences}},
  booktitle = {BMVC},
  year = {2025},
  url = {https://openreview.net/forum?id=twhgUxkQqQ},
}
Figure