Abstract
Contrastive learning applied to self-supervised representation learning has seen a resurgence in deep models. In this paper, we find that existing contrastive learning based solutions for self-supervised video recognition focus on inter-variance encoding but ignore the intra-variance existing in clips within the same video. We thus propose to learn dual representations for each clip which (i) encode intra-variance through a shuffle-rank pretext task; (ii) encode inter-variance through a temporal coherent contrastive loss. Experiment results show that our method plays an essential role in balancing inter and intra variances and brings consistent performance gains on multiple backbones and contrastive learning frameworks. Integrated with SimCLR and pretrained on Kinetics-400, our method achieves 82.0% and 51.2% downstream classification accuracy on UCF101 and HMDB51 test sets respectively and 46.1% video retrieval accuracy on UCF101, outperforming both pretext-task based and contrastive learning based counterparts.
Introduction: The Contrastive Paradox
In the world of self-supervised learning, contrastive learning is a dominant paradigm. The core idea is to take two clips from the same video and "pull" their features together in mathematical space while "pushing" clips from different videos apart.
However, this approach is fundamentally flawed for video data. A single video is not a static object; for instance, a video of a "High Jump" contains distinct mini-actions like running and jumping. If a model is forced to make the features for these distinct phases identical simply because they belong to the same video, it loses the ability to represent temporal nuances. This leads to "over-strong" instance discrimination—the model recognizes specific videos it has seen before but fails to generalize to new semantic concepts in downstream tasks. This paper addresses this "Contrastive Paradox" by balancing inter-video and intra-video variance.
Methodology: DualVar Framework
The authors propose a "Dual Representation" strategy to capture both the "forest" (the whole video) and the "trees" (the sub-clips).
- Dual Representations: Instead of encoding a clip into a single feature vector, the model projects it into two vectors ($q^1, q^2$), representing the two halves of the input clip.
- Shuffle-Rank Pretext Task: To teach the model about intra-video differences, sub-clips are shuffled and processed through a ranking loss. Unlike traditional "order prediction" (which is a binary classification), ranking enforces a small, flexible margin between sub-clip features. This allows the model to learn that sub-clips are different without completely shattering the video's global identity.
- Temporal Coherent Contrastive Loss: To maintain inter-video discrimination, the framework adapts standard contrastive losses (like SimCLR or MoCo). This ensures that all dual representations from the same video are still closer to each other than to those from entirely different videos.
By combining these elements, the final loss function forces the model to balance being "same-ish" (same video) and "different-ish" (different time steps).
Experiments and Key Results
The researchers evaluated their method using standard benchmarks: Kinetics-400 for large-scale pretraining, and UCF101 and HMDB51 for downstream fine-tuning and retrieval tasks.
As shown in the table above, the "DualVar" method (SR+TC) showed consistent gains across different backbones and frameworks. Key highlights include:
- Action Recognition: When integrated with SimCLR and pretrained on Kinetics-400, the model achieved 82.0% accuracy on UCF101 and 51.2% on HMDB51, outperforming state-of-the-art methods like RSPNet and VideoMoCo.
- Video Retrieval: In retrieval tasks, the model reached a Top-1 accuracy of 46.1% on UCF101, a significant jump over previous pretext-task based and contrastive learning counterparts.
- Ablation Success: The authors demonstrated that the "Shuffle-Rank" task alone could increase intra-variance by up to 14 times compared to standard SimCLR, proving the mechanism effectively preserves temporal structure.
Conclusion
By allowing for higher intra-variance, the pretrained encoder becomes more "flexible," making it much better at transferring knowledge to new datasets. While the performance relies on sensitive hyperparameters like the ranking loss temperature, the DualVar framework provides a robust solution to the limitations of standard instance discrimination in video self-supervised learning.
Summary updated