Abstract: Establishing correspondences between image pairs is a long-studied problem in computer vision. With recent large-scale foundation models showing strong zero-shot performance on downstream tasks, there has been interest in using their internal feature maps for semantic correspondence. Recent works observe that features from DINOv2 and Stable Diffusion (SD) are complementary: DINOv2 produces accurate but sparse correspondences, while SD produces spatially consistent ones. Consequently, current state-of-the-art methods ensemble both, which is computationally expensive. In this work, we replace SD features with a superior matching algorithm imbued with spatial consistency. Specifically, we replace standard nearest neighbors matching with an optimal transport algorithm including a Gromov-Wasserstein spatial smoothness prior. We significantly boost DINOv2 performance, rivaling or surpassing state-of-the-art SD-based methods while being 5–10x more efficient. Code: https://github.com/fsnelgar/semantic_matching_gwot.
Introduction: The Computational Bottleneck of Semantic Matching
Semantic correspondence involves finding matching parts between different instances of the same object class (e.g., mapping the ear of a golden retriever to the ear of a pug). This task is notoriously difficult due to variations in pose, scale, and non-rigid deformations.
Recently, researchers have achieved impressive results by extracting internal feature maps from large foundation models. DINOv2 provides highly accurate but spatially sparse matches, while Stable Diffusion (SD) provides excellent spatial consistency. To get the best of both worlds, current state-of-the-art (SOTA) approaches ensemble features from both models. However, running multiple billion-parameter models just for feature extraction is slow and memory-intensive. This paper asks: can we achieve the spatial consistency of Stable Diffusion without actually using it, relying solely on the lightweight DINOv2 model?
Methods: Upgrading the Matching Algorithm
Instead of upgrading the feature extractor, the authors upgrade the matching algorithm. Standard methods use simple Nearest Neighbor (NN) matching. This paper replaces NN with an Optimal Transport (OT) framework that explicitly encodes the desirable properties of semantic matching:
- Feature Similarity: Uses standard cosine similarity of DINOv2 features as the base cost.
- Spatial Smoothness (Gromov-Wasserstein): This is the core novelty. They use a Gromov-Wasserstein (GW) OT formulation, which penalizes matches where neighboring pixels in the source image map to distant pixels in the target image. To avoid the massive computational cost of standard GW matrix multiplications, they implement this using highly optimized 2D convolutions.
- Object Symmetry: Local features can be ambiguous (e.g., left vs. right wheels). They introduce a symmetry-aware objective that enforces horizontal ordering to respect the global pose.
- Unbalanced Formulation: Standard OT forces every pixel to match. Because of occlusions, they relax this constraint using a Kullback-Leibler (KL) divergence penalty, allowing the model to leave some pixels unmatched.
Key Results: Efficiency and Accuracy
The authors evaluate their method on three standard benchmarks: TSS (rigid objects), PF-PASCAL (pose/scale variation), and SPair-71k (large-scale, extreme variations). The results demonstrate that mathematical priors can effectively replace billions of parameters.
- Massive Efficiency Gains: The proposed method requires only 217ms per image pair and relies on an 87M parameter model. In contrast, SD-based SOTA methods (like DIFT or ToTF) take between 1,900ms and 2,300ms and require >1.2B parameters—an 8-10x speedup.
- State-of-the-Art Accuracy: On the TSS dataset, the method achieves a new SOTA (92.3% average PCK vs. the previous 88.7%).
- Highly Competitive: On the challenging SPair-71k dataset, the GWOT method boosts the DINOv2 baseline from 55.7% to 62.3%, closing the gap with heavy ensemble methods at a fraction of the cost.
Discussion and Limitations
The primary insight is that spatial consistency is a geometric property that can be enforced during matching rather than being "learned" by a generative model. However, limitations remain:
- Extreme Scale Changes: Fixed-radius spatial penalties can struggle when scale differences are extreme.
- Outlier Smoothing: If initial matches are overwhelmingly wrong, the GW prior can reinforce incorrect "smooth" regions.
- Symmetry Assumptions: The symmetry loss assumes moderate pose variations and may fail in extreme opposite-viewpoint comparisons.
Future work will explore adaptive filter sizes and extending this efficient framework to video label propagation tasks.
Summary updated