Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps

Octave Mariotti, Oisin Mac Aodha, Hakan Bilen

Introduction

Semantic Correspondence (SC) is a fundamental computer vision task that involves identifying functionally equivalent parts across different instances of the same object category—for example, matching the "front-left headlight" of a vintage sedan to that of a modern SUV. While recent self-supervised learning (SSL) models like DINOv2 have demonstrated remarkable feature representation capabilities, they often struggle with the underlying 3D geometry of objects.

Specifically, these models frequently suffer from two types of "hallucinations":

  1. Symmetry Collapse: Confusing the left and right sides of an object due to visual similarity.
  2. Part Confusion: Misidentifying repeated components, such as matching a front wheel in one image to a rear wheel in another because the model lacks spatial context.
Comparison of DINOv2 and the proposed method, highlighting how viewpoint-guided spherical maps resolve symmetry collapse and part confusion.

Viewpoint-Guided Spherical Maps

To address these geometric inconsistencies without the need for expensive 3D CAD models or dense manual annotations, this paper proposes a "middle ground" approach: Viewpoint-Guided Spherical Maps. The core idea is to learn a mapping function, $f_S$, that projects image pixels onto the surface of a virtual 3D unit sphere.

Technical Framework

The method utilizes a Spherical Prototype, a category-conditioned neural network that learns a generic representation of an object class on a sphere. This prototype attempts to reconstruct SSL features based on spherical coordinates, effectively anchoring 2D features in a 3D-aware space.

To ensure the spherical mapping is geometrically sound, three key regularization losses are employed:

  • Viewpoint Regularization: Leverages coarse viewpoint labels (e.g., "front-left") to align the spherical map's orientation with the camera's perspective.
  • Relative Distance Loss: A triplet loss that prevents spatial "collapse" by ensuring pixels distant in the 2D image remain distant on the 3D sphere.
  • Orientation Loss: Preserves the relative spatial ordering of parts (e.g., left-to-right) during the projection process, preventing mirrored matches.

Experimental Results

The proposed method was primarily evaluated on the SPair-71k benchmark, which includes 18 diverse object categories. It was also tested on the AwA (Animals with Attributes) dataset to evaluate generalization to unseen classes.

Performance and Efficiency

The integration of spherical maps significantly boosted DINOv2's performance, particularly for symmetric categories. For instance, accuracy improved by 21.0% for buses and 15.7% for cars on SPair-71k. Beyond accuracy, the method is highly efficient. Compared to feature-heavy approaches like those using Stable Diffusion, this spherical mapping is approximately 10x faster, processing 3.3 pairs per second.

Rethinking Metrics: Keypoint Average Precision (KAP)

The authors argue that the standard Percentage of Correct Keypoints (PCK) metric is insufficient because it only evaluates visible points. They introduce Keypoint Average Precision (KAP), which treats correspondence as a retrieval task. KAP more effectively penalizes "spurious" matches where a model incorrectly links a visible part to a hidden one on the opposite side of the object.

Conclusion and Future Work

By using a sphere as a universal geometric prior, this research provides a simple yet effective way to inject 3D common sense into 2D vision models. While the approach excels for objects homeomorphic to a sphere (like most animals and vehicles), challenges remain for objects with complex topologies (like chairs) or high deformability. Future research will focus on removing the requirement for foreground segmentation masks during training, moving toward a fully unsupervised 3D-aware correspondence framework.

Summary updated

Abstract

Recent self-supervised models produce visual features that are not only effective at encoding image-level, but also pixel-level, semantics. They have been reported to obtain impressive results for dense visual semantic correspondence estimation, even outperforming fully-supervised methods. Nevertheless, these models still fail in the pres-ence of challenging image characteristics such as symme-tries and repeated parts. To address these limitations, we propose a new semantic correspondence estimation method that supplements state-of-the-art self-supervised features with 3D understanding via a weak geometric spherical prior. Compared to more involved 3D pipelines, our model provides a simple and effective way of injecting informative geometric priors into the learned representation while requiring only weak viewpoint information. We also propose a new evaluation metric that better accounts for re-peated part and symmetry-induced mistakes. We show that our method succeeds in distinguishing between symmetric views and repeated parts across many object categories in the challenging SPair-71 k dataset and also in generalizing to previously unseen classes in the AwA dataset.
BibTeX
@inproceedings{mariotti2023improving,
  author = {Octave Mariotti and Oisin Mac Aodha and Hakan Bilen},
  title = {{Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure