Mining Better Samples for Contrastive Learning of Temporal Correspondence

Sangryul Jeon, Dongbo Min, Seungryong Kim, Kwanghoon SohnCVPR 2021 · OpenReview

Abstract

We present a novel framework for contrastive learning of pixel-level representation using only unlabeled video. Without the need of ground-truth annotation, our method is capable of collecting well-defined positive correspondences by measuring their confidences and well-defined negative ones by appropriately adjusting their hardness during training. This allows us to suppress the adverse impact of ambiguous matches and prevent a trivial solution from being yielded by too hard or too easy negative samples. To accomplish this, we incorporate three different criteria that ranges from a pixel-level matching confidence to a video-level one into a bottom-up pipeline, and plan a curriculum that is aware of current representation power for the adaptive hardness of negative samples during training. With the proposed method, state-of-the-art performance is attained over the latest approaches on several video label propagation tasks.

Introduction: The Challenge of Temporal Correspondence

Learning how to track specific pixels across video frames—known as temporal correspondence—is fundamental for tasks like object tracking, video editing, and autonomous driving. Traditionally, this requires massive amounts of human-annotated data, which is incredibly expensive and difficult to produce for every pixel in every frame.

Self-supervised learning offers a "free" alternative by using the video itself as supervision. Previous methods usually relied on reconstruction loss (trying to predict a pixel's color in a future frame). However, this fails when objects are occluded or lighting changes. More recent contrastive learning approaches (learning to distinguish a "true" match from "false" ones) are more robust but struggle with sample quality: if the "false" examples are too easy, the model learns nothing; if they are too ambiguous, the model gets confused.

Comparison of different loss functions for temporal correspondence learning. Our method (c) focuses on mining samples with an appropriate hardness range.

Methodology: Mining Better Samples

The authors propose a framework that "mines" better samples for contrastive learning through two main components: a bottom-up pipeline for positives and a dynamic curriculum for negatives.

Overview of the proposed framework, showing the query/key image processing and the mining stages for positive and negative correspondences.

1. Mining Positive Correspondences (The Bottom-Up Pipeline)

To find a reliable "positive" match without labels, the system passes potential matches through three filters:

  • Forward-Backward Consistency: It checks if a pixel tracked from Frame A to Frame B can be tracked back to its original position in Frame A.
  • Optimal Transport (OT): It solves a global optimization problem (using the Sinkhorn algorithm) to ensure that pixels aren't all trying to match to the same "easy" spot, enforcing a more realistic one-to-one mapping across the whole image.
  • Temporal Coherence: It restricts matches to a local spatial window that grows over time, filtering out visually similar but distant background noise.

2. Mining Negative Correspondences (The Dynamic Curriculum)

Instead of comparing a pixel against every other pixel in the image, the authors select "semi-hard" negatives.

  • Hardness Control: They ignore pixels that are too easy to distinguish and those that are so similar they might actually be the same object (which would confuse the model).
  • Dynamic Curriculum: The "hardness" threshold isn't fixed. The model measures its own confidence; as it gets better at the task (lower variance in its confidence scores), it automatically starts picking harder negative samples to challenge itself further.

Key Results

The proposed method set new state-of-the-art benchmarks for self-supervised correspondence across several benchmarks including DAVIS-2017, YouTube-VOS, and JHMDB:

  • DAVIS-2017: Achieved a 70.3 $\mathcal{J}\&\mathcal{F}$ mean, significantly outperforming the previous best self-supervised method, CRW (67.6), and even beating the supervised ResNet-18 baseline (62.9).
  • YouTube-VOS: Showed a large gain over reconstruction-based methods like MAST (+3.1 points).
  • Human Tracking: On the JHMDB pose tracking task, it achieved a 60.5 PCK (at $\alpha=0.1$), proving that the pixel-level features are fine-grained enough to track small human joints accurately.

Conclusion and Limitations

This research reinforces the idea that in contrastive learning, the quality of negative samples is just as important as the positives. By avoiding "too easy" and "too hard" samples, the model avoids gradient vanishing and noisy updates.

However, limitations remain. The temporal coherence filter relies on a local window; if an object moves extremely fast, the model might fail to find the correct positive match. Additionally, the multi-stage bottom-up mining process is more computationally demanding than simple reconstruction-based self-supervision. Future work may extend this "sample mining" philosophy to other domains of self-supervised representation learning.

Summary updated

Abstract

We present a novel framework for contrastive learning of pixel-level representation using only unlabeled video. Without the need of ground-truth annotation, our method is capable of collecting well-defined positive correspondences by measuring their confidences and well-defined negative ones by appropriately adjusting their hardness during training. This allows us to suppress the adverse impact of ambiguous matches and prevent a trivial solution from being yielded by too hard or too easy negative samples. To accomplish this, we incorporate three different criteria that ranges from a pixel-level matching confidence to a video-level one into a bottom-up pipeline, and plan a curriculum that is aware of current representation power for the adaptive hardness of negative samples during training. With the proposed method, state-of-the-art performance is attained over the latest approaches on several video label propagation tasks.
BibTeX
@inproceedings{jeon2021mining,
  author = {Sangryul Jeon and Dongbo Min and Seungryong Kim and Kwanghoon Sohn},
  title = {{Mining Better Samples for Contrastive Learning of Temporal Correspondence}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=BxQxFgj8xJmp},
}
Figure