TL;DR
State-of-the-art semantic correspondence methods typically ensemble DINOv2 and Stable Diffusion features to achieve both precise and spatially consistent matches, but this approach is computationally heavy. This paper introduces a highly efficient alternative by replacing Stable Diffusion entirely with a Fused Gromov-Wasserstein optimal transport matching algorithm. By encoding spatial smoothness, object symmetry, and scale-handling directly into the optimization objective, they boost the performance of a single DINOv2 model to competitive levels while running 5–10x faster.
What Problem Does This Paper Address?
Establishing semantic correspondences—finding matching pixels between different instances of the same object class (e.g., the ear of a Siamese cat vs. the ear of a tabby cat)—is a foundational challenge in computer vision. It is exceptionally difficult due to intra-class variations, non-rigid deformations, and changes in camera pose.
Recently, researchers discovered that features from large foundation models are highly complementary for this task: DINOv2 provides accurate but sparse matches, while Stable Diffusion (SD) features offer excellent spatial consistency. Consequently, current top-performing methods ensemble both models. However, extracting feature maps from massive diffusion models is incredibly slow and memory-intensive. The authors of this paper tackle this bottleneck by asking a smart question: Can we achieve spatial consistency through a superior matching algorithm instead of relying on a heavier model architecture?
Methods
Instead of standard nearest-neighbors matching, the authors formulate the matching process as a Fused Gromov-Wasserstein (GW) Optimal Transport problem. They use a single, frozen DINOv2 (ViT-B/16) model as the feature extractor and solve a comprehensive optimization problem that explicitly encodes the desirable properties of semantic matching:
- Feature Similarity (Ground Cost): Measures the semantic similarity between patches using the cosine similarity of DINOv2 features.
- Spatial Smoothness (Gromov-Wasserstein Prior): A GW term penalizes neighboring patches in the source image from matching to distant patches in the target image. Crucially, because these local neighborhood matrices are sparse, the authors implement this term using fast 2D convolutions, avoiding expensive matrix multiplications.
- Object Symmetry: Many objects share a vertical axis of symmetry, which often confuses standard matchers (e.g., swapping left and right wheels). The authors introduce a symmetry-aware objective that penalizes matches that violate horizontal ordering consistency.
- Unbalanced Formulation: Strict optimal transport assumes mass conservation (one-to-one mapping). To handle objects of different scales or regions that are occluded, the authors relax this constraint into a partially unbalanced problem using a Kullback-Leibler (KL) divergence penalty.
The entire objective is solved efficiently at test time using projected gradient descent over 50 steps.
Datasets
The method was evaluated on three standard semantic correspondence benchmarks:
- TSS (Taniai et al.): 400 image pairs primarily featuring vehicles. It contains rigid structures with low pose variation, making it an excellent testbed for strong spatial smoothness priors.
- PF-PASCAL: 1,351 image pairs across 20 object categories, presenting more challenging variations in pose and scale.
- SPair-71k: A massive and highly difficult dataset containing 70,958 image pairs across 18 categories, characterized by significant viewpoint changes, non-rigid deformations, and heavy scene clutter.
Key Insights and Ideas
- Algorithm over Architecture: Instead of throwing more parameters (Stable Diffusion) at spatial noise, fixing the mathematical formulation of the matcher yields a massive efficiency boost with virtually no loss in accuracy.
- Exploiting Sparsity for Speed: Gromov-Wasserstein distance is notoriously expensive to compute. By restricting the spatial smoothness prior to a local radius ($\delta_{min}, \delta_{max}$), the cost matrices become sparse, allowing the optimization step to leverage highly optimized deep learning primitives (2D convolutions).
- Geometry via Optimization: Rather than training a geometric warping network to understand object orientation, enforcing horizontal coordinate ordering mathematically within the transport problem successfully resolves left-vs-right ambiguities.
Key Results
Computational Efficiency
The proposed method drastically reduces the computational footprint of zero-shot semantic matching. It cuts down the parameter count from ~1.3 Billion (for SD ensembles) to just 87 Million (DINOv2 alone). As a result, the total time per image pair drops from ~2 seconds to just 217 milliseconds, representing a 5–10x speedup.
Matching Accuracy
- TSS Dataset: The method achieves state-of-the-art performance with a 92.3% PCK average, outperforming both zero-shot and fully supervised ensembled methods.
- PF-PASCAL & SPair-71k: The algorithm significantly boosts the baseline DINOv2 performance (e.g., from 55.7% to 62.3% PCK on SPair-71k) and performs competitively with Stable Diffusion ensembles, even surpassing them in categories with rigid geometries.
Limitations and Future Work
While powerful, the method relies on the assumption that correspondences are locally spatially consistent. The authors note several failure modes:
- Extreme Scale/Pose Changes: Fixed neighborhood thresholds ($\delta_{min}, \delta_{max}$) fail when scale differences are extreme or when an object is viewed from completely opposite sides.
- Outlier Propagation: If the underlying DINOv2 features are severely degraded, the spatial smoothness constraint can inadvertently smooth out incorrect matches, creating clusters of plausible-looking but wrong correspondences.
- Resolution Constraints: Because the method operates on patch-level tokens, fine-grained keypoints on very small objects can collapse into the same patch.
Future Directions: The authors suggest exploring adaptive filter sizes for the Gromov-Wasserstein formulation to handle scale variations dynamically, extending the framework to higher-order interactions, and applying it to video label propagation tasks.
Summary updated