Abstract
Existing datasets for semantic correspondence are often limited in terms of both the amount of labeled data and diversity of labeled keypoints due to the tremendous cost of manual correspondence labeling. To address this issue, we propose the first self-supervised learning framework that utilizes a large amount of web videos collected and annotated fully automatically. Our main motivation is that smooth changes between consecutive video frames allow to build accurate space-time correspondences with no human intervention. Hence, we establish space-time correspondences within each web video and leverage them for deriving pseudo correspondence labels between two distant frames of the video. In addition, we present a dedicated training strategy that facilitates stable training using web videos with such pseudo labels. Our experiments on public benchmarks demonstrated that the proposed method surpasses existing self-supervised learning models and that our self-supervised learning as pretraining for supervised learning improves performance substantially.
Introduction
Semantic correspondence is the task of finding matching points between two different images of the same category—such as identifying the same structural parts on two different models of cars or two different breeds of dogs.
The primary challenge in this field is the annotation bottleneck. Manually labeling point-to-point correspondences across thousands of image pairs is incredibly tedious and expensive. Consequently, existing datasets are small and lack diversity. While self-supervised methods exist, they often rely on synthetic transformations (like warping a single image), which fail to capture the complex real-world variations in lighting, pose, and appearance that occur between different individuals of the same species or different products in the same category.
Methodology
The proposed framework turns web videos into a massive training ground for semantic matching through a fully automatic three-stage pipeline.
1. Automatic Data Collection and Pseudo-Label Generation
The system crawls YouTube for specific object classes, using thumbnail classification to filter for relevant content. Once relevant videos are downloaded, "shot detection" splits them into clips to ensure visual continuity. Within these clips, the model tracks pixels across consecutive frames using a space-time correspondence encoder. By chaining these tracks, the system establishes correspondences between temporally distant frames, creating "pseudo-labels."
To ensure the quality of these labels, the authors employ an Isolation Forest algorithm to detect and remove "noisy" tracks—those that do not follow the general motion of the object. This outlier detection is crucial, as tracking errors can otherwise lead to "drift" in the training data.
2. Training with Domain Adaptation
The model, based on the CATs architecture, is trained using these generated pseudo-labels. However, web videos often differ significantly in quality and composition from standard benchmark photos. To bridge this gap, the authors employ a Domain Adversarial Learning strategy.
This strategy uses a discriminator to force the network to learn features that are "domain-invariant," ensuring the model performs reliably on both raw YouTube clips and high-quality evaluation images.
Datasets and Evaluation
The researchers curated a new Web Video Dataset containing 2,124 videos, resulting in 36,879 clips and over 2.8 million frame pairs. Evaluation was conducted on three standard benchmarks:
- PF-PASCAL: 1,351 image pairs across 20 categories.
- PF-Willow: 900 image pairs across 4 categories.
- SPair-71K: A large-scale benchmark with 70,958 pairs featuring diverse viewpoints and scales.
Key Results
The proposed method achieved state-of-the-art (SOTA) results across both self-supervised and supervised categories:
- Self-Supervised Leader: On the difficult SPair-71K dataset, the model achieved 27.3% PCK (Percentage of Correct Keypoints), significantly outperforming previous self-supervised methods like CNNGeo (20.6%) and A2Net (22.3%).
- Supervised Boost: When used as a pre-training step for supervised models, performance improved substantially. On SPair-71K, the transferred model reached 54.0% PCK, surpassing the previous best of 53.7% (TransforMatcher).
- Efficiency: The authors found that simple thumbnail filtering is a highly efficient way to curate relevant training content without downloading massive amounts of irrelevant data.
Limitations and Future Work
While the method is robust, two main failure cases for pseudo-labeling were identified:
- Iterative Textures: Repeating patterns (like fences or tiles) can confuse the tracking model.
- Featureless Areas: Textureless regions (like clear skies) lack unique points for reliable tracking.
Future research may involve developing algorithms to correct these tracking errors by incorporating global context or 3D geometric constraints.
Summary updated