Abstract
Self-supervised approaches for video have shown impressive results in video understanding tasks. However, unlike early works that leverage temporal self-supervision, current state-of-the-art methods primarily rely on tasks from the image domain (e.g., contrastive learning) that do not explicitly promote the learning of temporal features. We identify two factors that limit existing temporal self-supervision: 1) tasks are too simple, resulting in saturated training performance, and 2) we uncover shortcuts based on local appearance statistics that hinder the learning of high-level features. To address these issues, we propose 1) a more challenging reformulation of temporal self-supervision as frame-level (rather than clip-level) recognition tasks and 2) an effective augmentation strategy to mitigate shortcuts. Our model extends a representation of single video frames, pre-trained through contrastive learning, with a transformer that we train through temporal self-supervision. We demonstrate experimentally that our more challenging frame-level task formulations and the removal of shortcuts drastically improve the quality of features learned through temporal self-supervision. Our extensive experiments show state-of-the-art performance across 10 video understanding datasets, illustrating the generalization ability and robustness of our learned video representations.
Introduction
In self-supervised learning (SSL) for video, the primary goal is to learn temporal dynamics—how objects move and evolve over time—without manual labels. Despite the potential, recent state-of-the-art methods have shifted away from temporal-specific tasks toward image-based contrastive learning, which prioritizes spatial appearance.
A significant hurdle identified in this work is the presence of "shortcuts." Even when temporal tasks are employed (e.g., determining if a clip is in order), models often "cheat" by exploiting local appearance statistics. For instance, a model might match a small patch of static background pixels across frames to verify order, rather than understanding complex motions like a person's stride. This leads to high training accuracy but fails to produce high-level features useful for downstream video understanding.
Methods
To overcome these limitations, we introduce a two-pronged strategy designed to break shortcuts and increase the complexity of temporal learning.
Breaking Shortcuts with Frame-wise Augmentation
Standard video SSL typically applies identical augmentations (e.g., the same crop or color jitter) to all frames in a clip, preserving low-level pixel correlations that models can exploit. We propose independent frame-wise augmentation, where spatial jittering (random resizing and cropping) is applied uniquely to every frame. This prevents the model from relying on local pixel matches and forces it to focus on global scene context, such as human pose or object deformation.
Challenging Frame-level Pretext Tasks
We move beyond simple clip-level labels to more granular frame-level recognition tasks:
- Out-of-order Frame Localization (OFL): The model must pinpoint specific frames within a sequence that have been shuffled or shifted.
- Time-varying Skiprate Prediction (TSP): The model predicts the playback speed (e.g., 1x, 4x, 8x) between every consecutive pair of frames in a sequence with dynamic speed changes.
These tasks are implemented using a Video Transformer Network (VTN), which integrates a Vision Transformer (ViT) image encoder with a temporal transformer.
Experimental Results
We pre-trained our model on the unlabeled videos of Kinetics400 and evaluated its performance across 10 diverse datasets.
Performance Benchmarks
Our approach achieved state-of-the-art (SOTA) results on nearly all benchmarks:
- Video Retrieval: On UCF101, we achieved 85.1% R@1, a 3.3% improvement over the previous best, VideoMAE.
- Temporal-Heavy Datasets: On Something-Something V2 (SSv2), which demands deep motion understanding, our model improved linear probing accuracy by 3.2% over prior methods.
- Gait and Pose: In Gait Recognition on CASIA-B, our model reached 73.28% mean accuracy, significantly outperforming VideoMAE’s 47.97%.
Analysis of Shortcuts
To validate our "shortcut" hypothesis, we trained a model on tiny 32x32 pixel patches. With standard augmentations, this "blind" model achieved ~78% accuracy on temporal tasks. However, with our proposed frame-wise augmentations, accuracy dropped to ~26%, confirming that the model was previously "cheating" and is now forced to learn more meaningful features.
Discussion and Conclusion
By increasing task difficulty and removing shortcuts, we significantly enhance the quality of features learned through temporal self-supervision. Our model demonstrates remarkable robustness to noise and generalizes well across tasks ranging from action recognition to object segmentation.
While our evaluation primarily focuses on fixed-feature performance (linear probing and retrieval) to highlight representation quality, future work could explore end-to-end fine-tuning. Additionally, the "anti-shortcut" strategies presented here could be extended to other temporal domains, such as multi-modal video or time-series data.
Summary updated