Abstract
Self-supervised approaches for video have shown impressive results in video understanding tasks. However, unlike early works that leverage temporal self-supervision, current state-of-the-art methods primarily rely on tasks from the image domain (e.g., contrastive learning) that do not explicitly promote the learning of temporal features. We identify two factors that limit existing temporal self-supervision: 1) tasks are too simple, resulting in saturated training performance, and 2) we uncover shortcuts based on local appearance statistics that hinder the learning of high-level features. To address these issues, we propose 1) a more challenging reformulation of temporal self-supervision as frame-level (rather than clip-level) recognition tasks and 2) an effective augmentation strategy to mitigate shortcuts. Our model extends a representation of single video frames, pre-trained through contrastive learning, with a transformer that we train through temporal self-supervision. We demonstrate experimentally that our more challenging frame-level task formulations and the removal of shortcuts drastically improve the quality of features learned through temporal self-supervision. The generalization capability of our self-supervised video method is evidenced by its state-of-the-art performance in a wide range of high-level semantic tasks, including video retrieval, action classification, and video attribute recognition, as well as low-level temporal correspondence tasks like video object segmentation and pose tracking. Additionally, we show that the video representations learned through our method exhibit increased robustness to the input perturbations.
Introduction
Self-supervised learning (SSL) has revolutionized image understanding by enabling models to learn from unlabeled data. However, in the video domain, SSL progress has plateaued. Most state-of-the-art methods treat video frames as independent images, largely ignoring temporal dynamics. This paper identifies two critical roadblocks: the "shortcut problem," where models exploit local pixel statistics instead of learning motion, and "easy pretext tasks" that lead to early training saturation. By introducing independent frame-wise augmentations and more difficult frame-level pretext tasks, the authors force models to learn true temporal dynamics, achieving state-of-the-art performance across various video tasks.
Methodology
The proposed strategy addresses the limitations of current temporal self-supervision through two main innovations: frame-wise augmentation and frame-level pretext tasks.
Frame-wise Augmentation
Standard video SSL applies the same spatial augmentations (e.g., cropping, color jitter) to all frames in a clip. This consistency creates a shortcut: models can determine frame order by simply matching local background patches. To break this, the authors propose independent frame-wise augmentation, where each frame receives a unique spatial transformation. This destroys local pixel-matching shortcuts, forcing the model to rely on global scene context and object dynamics.
Frame-level Pretext Tasks
The authors shift from clip-level classification to more granular frame-level recognition tasks:
- Out-of-order Frame Localization (OFL): Instead of identifying if a clip is shuffled, the model must pinpoint the specific out-of-place frames.
- Time-varying Skiprate Prediction (TSP): The model predicts the specific playback speed (e.g., 1x, 4x, 8x) between every consecutive pair of frames.
The architecture utilizes a Video Transformer Network (VTN), combining a Vision Transformer (ViT) for frame encoding with a temporal transformer for sequence processing.
Experimental Setup
The model was pre-trained on the Kinetics400 dataset without labels. Evaluation spanned several downstream tasks and datasets:
- Action Recognition: UCF101, HMDB51, Something-Something V2 (SSv2), NTU60.
- Holistic Understanding: HVU (scenes, objects, events).
- Temporal Correspondence: DAVIS-2017 (segmentation), JHMDB (pose tracking).
- Gait Recognition: CASIA-B.
- Video Retrieval: UCF101 and HMDB51.
Discussion: Key Insights
- Task Difficulty Matters: Moving to frame-level tasks reduced pretext accuracy (from ~99% to ~87%) but significantly boosted downstream performance. Harder tasks prevent early saturation and encourage deeper feature learning.
- Global Semantics: Independent augmentations force the model to understand "global" semantics—such as human limb movement or object deformation—rather than relying on "pixel tracking."
- Temporal Supervision Value: While static image features are strong, the "No-Shortcuts" temporal supervision consistently adds 4-7% accuracy across tasks, highlighting the importance of explicit temporal learning.
Results
The method demonstrates significant improvements over previous self-supervised SOTA:
- Video Retrieval: Achieved absolute improvements of 3.3% (UCF101) and 8.2% (HMDB51) in Top-1 accuracy, outperforming methods like SVT and VideoMAE.
- Action Recognition: On the temporally-intensive SSv2 dataset, the model surpassed previous linear probing benchmarks by 3.2%.
- Robustness: The model shows increased resilience to input perturbations like Gaussian noise or JPEG compression, indicating it learns robust structural features rather than fragile pixel patterns.
Conclusion
This work demonstrates that realizing the potential of temporal self-supervision requires moving beyond simple tasks and eliminating easy shortcuts. While the current evaluation focused on fixed-feature backbones for scalability, future work could explore full fine-tuning and the application of these "no-shortcut" principles to other time-series modalities like audio or sensor data.
Summary updated