Abstract
Learning temporal correspondence from unlabeled videos is of vital importance in computer vision, and has been tackled by different kinds of self-supervised pretext tasks. For the self-supervised learning, recent studies suggest using large-scale video datasets despite the training cost. We propose a spatial-then-temporal pretext task to address the training data cost problem. The task consists of two steps. First, we use contrastive learning from unlabeled still image data to obtain appearance-sensitive features. Then we switch to unlabeled video data and learn motion-sensitive features by reconstructing frames. In the second step, we propose a global correlation distillation loss to retain the appearance sensitivity learned in the first step, as well as a local correlation distillation loss in a pyramid structure to combat temporal discontinuity. Experimental results demonstrate that our method surpasses the state-of-the-art self-supervised methods on a series of correspondence-based tasks. The conducted ablation studies verify the effectiveness of the proposed two-step task and loss functions.
Introduction
Video correspondence—the ability to match a specific pixel or object across different frames—is a foundational requirement for computer vision tasks like optical flow, object segmentation, and point tracking. Current self-supervised approaches typically focus on either spatial-discriminative features (learning from static images to distinguish objects) or temporal-repetitive features (learning from video to track motion).
Spatial-only models are robust to appearance changes but are easily confused by "distractors"—objects that look similar to the target. Conversely, temporal-only models excel at following motion but often fail during dramatic deformations, occlusions, or significant appearance changes. The challenge lies in creating a representation that captures the strengths of both without one degrading the other during training. This paper introduces a two-stage self-supervised framework that achieves state-of-the-art video correspondence by synergizing spatial appearance features with temporal motion cues.
Methodology
The proposed Spatial-then-Temporal learning pipeline consists of two distinct phases designed to integrate appearance and motion sensitivity.
Step 1: Spatial Feature Learning
The model is first pre-trained on unlabeled images from ImageNet using contrastive learning (specifically the DetCo algorithm). This phase teaches the encoder to create discriminative representations based on appearance, effectively learning "what" an object looks like.
Step 2: Temporal Feature Learning
The model then transitions to unlabeled video data (YouTube-VOS) for a frame reconstruction task. To optimize this learning and maintain the spatial knowledge, three components are introduced:
- Pyramid Frame Reconstruction: The model reconstructs frames at multiple resolution levels within the neural network to capture multi-scale features.
- Local Correlation Distillation (LCD): To address temporal discontinuity where pixels might disappear due to downsampling, high-resolution feature maps serve as "pseudo-labels" to guide coarser, lower-resolution maps.
- Global Correlation Distillation (GCD): To prevent "catastrophic forgetting" of the spatial knowledge from Step 1, the original spatial-only model acts as a teacher. The student model is forced to maintain global matching capabilities similar to the teacher while acquiring new temporal cues.
- Entropy-based Selection: The distillation process is focused on regions with high uncertainty using an entropy-based mask, targeting areas where temporal tracking is most likely to fail.
Experimental Results
The method was evaluated on several downstream tasks, including semi-supervised video object segmentation (DAVIS-2017), human part propagation (VIP), and pose keypoint tracking (JHMDB).
Quantitative Performance
The proposed method sets new benchmarks for self-supervised video correspondence:
- DAVIS-2017: Achieved a score of 74.1% (J&F mean), outperforming previous state-of-the-art methods like VFS (68.9%) and LIIR (72.1%).
- VIP: Reached 41.0 mIoU, surpassing even some specialized supervised methods.
- JHMDB: Achieved 63.1 [email protected], demonstrating high accuracy in tracking fine-grained joint locations.
Qualitative Analysis
Visualizations demonstrate that the model successfully ignores distractors (e.g., a second person in the background) that typically confuse purely spatial models. Simultaneously, it maintains a lock on objects during fast motion, a scenario where purely temporal models often fail. The synergy of spatial and temporal features allows for more robust tracking through deformations and occlusions.
Conclusion
This work demonstrates that spatial and temporal cues are complementary. By employing a two-stage training process with distillation losses, the model achieves a synergy that enhances correspondence accuracy. While the sequential training process is effective, future research could explore simultaneous learning of these features and further improvements in data efficiency and handling extreme occlusions.
Summary updated