Abstract
Semantic correspondence aims to establish matches between semantically similar points across images with varying appearances and poses. This paper provides the first comprehensive survey and unified benchmarking of the field. Through systematic evaluation, the authors reveal that the backbone architecture is the most critical factor for success. Leveraging powerful foundation models like DINOv2, they propose a simple baseline that outperforms complex, specialized architectures, setting a new state-of-the-art.
Introduction
In computer vision, finding correspondences between images is a fundamental task. While geometric matching focuses on identical physical points and optical flow tracks motion, semantic correspondence matches points sharing the same meaning—such as the eyes of different cats in diverse environments.
Historically, the field has been fragmented, with researchers utilizing disparate backbones, resolutions, and training techniques. This lack of standardization made it difficult to discern whether improvements stemmed from architectural innovations or superior feature extractors. This paper brings order to the chaos with a clear taxonomy and a rigorous "apples-to-apples" comparison of existing methods.
Taxonomy of Semantic Correspondence
The authors categorize a decade of research into three primary pillars:
- Handcrafted Descriptors: Early techniques relying on SIFT or HOG.
- Architectural Improvements: This includes feature enhancement (e.g., hypercolumns) and matching refinement (e.g., cost volumes, flow fields, or geometric transformations).
- Training Strategies: The evolution from strong supervision (keypoint labels) to weak supervision (masks/image labels) and self-supervision (cycle consistency).
The "Backbone is King" Paradigm
A central finding of this study is that the most significant performance gains in recent years are attributable to the adoption of Vision Foundation Models (VFMs) like DINOv2 and Stable Diffusion. These models capture high-level semantics far more effectively than traditional CNNs.
The authors' proposed "Strong Baseline" reflects this insight. It utilizes a powerful foundation model (DINOv2), a standard ResNet bottleneck for feature enhancement, and a Neighborhood Consensus (NC) aggregator for match refinement. This straightforward approach demonstrates that feature adaptation is often more effective than complex, computationally expensive aggregation modules.
Experimental Results
The baseline was evaluated across several key benchmarks, including SPair-71k, PF-PASCAL, and AP-10K. On SPair-71k, the supervised model achieved a [email protected] of 85.2%, surpassing previous complex models like GeoAware-SC (82.9%). In zero-shot settings, combining Stable Diffusion and DINOv2 features yielded a 60.4% [email protected] without task-specific training.
The study also highlights that for Transformer-based models like DINOv2, higher image resolution consistently improves localization, whereas older CNNs often suffer from a loss of global context at higher resolutions due to fixed receptive fields.
Conclusion and Future Work
This research serves as a definitive guide for semantic matching, emphasizing the importance of the backbone architecture. While the proposed baseline sets a new standard, challenges remain regarding the computational cost of high-resolution cost volumes and the reliance on expensive keypoint labels for fine-tuning. Future work may explore more sophisticated adaptation techniques to further unlock the potential of foundation models.
Summary updated