Introduction
Semantic Correspondence (SC) is a fundamental computer vision task that involves identifying functionally equivalent parts across different instances of the same object category—for example, matching the "front-left headlight" of a vintage sedan to that of a modern SUV. While recent self-supervised learning (SSL) models like DINOv2 have demonstrated remarkable feature representation capabilities, they often struggle with the underlying 3D geometry of objects.
Specifically, these models frequently suffer from two types of "hallucinations":
- Symmetry Collapse: Confusing the left and right sides of an object due to visual similarity.
- Part Confusion: Misidentifying repeated components, such as matching a front wheel in one image to a rear wheel in another because the model lacks spatial context.
Viewpoint-Guided Spherical Maps
To address these geometric inconsistencies without the need for expensive 3D CAD models or dense manual annotations, this paper proposes a "middle ground" approach: Viewpoint-Guided Spherical Maps. The core idea is to learn a mapping function, $f_S$, that projects image pixels onto the surface of a virtual 3D unit sphere.
Technical Framework
The method utilizes a Spherical Prototype, a category-conditioned neural network that learns a generic representation of an object class on a sphere. This prototype attempts to reconstruct SSL features based on spherical coordinates, effectively anchoring 2D features in a 3D-aware space.
To ensure the spherical mapping is geometrically sound, three key regularization losses are employed:
- Viewpoint Regularization: Leverages coarse viewpoint labels (e.g., "front-left") to align the spherical map's orientation with the camera's perspective.
- Relative Distance Loss: A triplet loss that prevents spatial "collapse" by ensuring pixels distant in the 2D image remain distant on the 3D sphere.
- Orientation Loss: Preserves the relative spatial ordering of parts (e.g., left-to-right) during the projection process, preventing mirrored matches.
Experimental Results
The proposed method was primarily evaluated on the SPair-71k benchmark, which includes 18 diverse object categories. It was also tested on the AwA (Animals with Attributes) dataset to evaluate generalization to unseen classes.
Performance and Efficiency
The integration of spherical maps significantly boosted DINOv2's performance, particularly for symmetric categories. For instance, accuracy improved by 21.0% for buses and 15.7% for cars on SPair-71k. Beyond accuracy, the method is highly efficient. Compared to feature-heavy approaches like those using Stable Diffusion, this spherical mapping is approximately 10x faster, processing 3.3 pairs per second.
Rethinking Metrics: Keypoint Average Precision (KAP)
The authors argue that the standard Percentage of Correct Keypoints (PCK) metric is insufficient because it only evaluates visible points. They introduce Keypoint Average Precision (KAP), which treats correspondence as a retrieval task. KAP more effectively penalizes "spurious" matches where a model incorrectly links a visible part to a hidden one on the opposite side of the object.
Conclusion and Future Work
By using a sphere as a universal geometric prior, this research provides a simple yet effective way to inject 3D common sense into 2D vision models. While the approach excels for objects homeomorphic to a sphere (like most animals and vehicles), challenges remain for objects with complex topologies (like chairs) or high deformability. Future research will focus on removing the requirement for foreground segmentation masks during training, moving toward a fully unsupervised 3D-aware correspondence framework.
Summary updated