How Incomplete is Contrastive Learning? An Inter-intra Variant Dual Representation Method for Self-supervised Video Recognition

Lin Zhang, Qi She, Zhengyang Shen, Changhu Wang

Abstract

Self-supervised learning (SSL) has significantly advanced video recognition by leveraging unlabeled data through contrastive learning. However, current paradigms often over-optimize for instance discrimination, treating all clips from a single video as identical. This approach ignores the semantic evolution within a video, leading to an "incomplete" representation. We present DualVar, a framework that balances inter-video identity with intra-video variance. By utilizing dual representations and a Shuffle-Rank pretext task, DualVar captures both the global identity and the local temporal dynamics of video content. Our method achieves state-of-the-art performance on major benchmarks, including UCF101 and HMDB51.

Introduction

The dominant strategy in self-supervised video recognition has been contrastive learning, where models are trained to recognize that clips from the same video are similar while clips from different videos are distinct. While effective, this method often forces a model to treat every phase of an action—such as the approach, plant, and jump in a high jump video—as semantically identical. This suppression of intra-video variance can limit a model's ability to understand the nuances of motion, resulting in a brittle feature space.

The Problem of Incomplete Contrastive Learning

Standard contrastive learning frameworks like MoCo and SimCLR focus heavily on instance discrimination. By pushing the variance between clips of the same video toward zero, they inadvertently discard temporal information that is crucial for downstream tasks. The authors of DualVar argue that a "Goldilocks zone" exists where a model maintains video-level identity while simultaneously recognizing that different moments in time represent distinct semantic states. Finding this balance is key to creating more robust and generalizable video representations.

Methodology: The DualVar Framework

To address these limitations, the authors introduce DualVar, which consists of three core innovations:

  1. Dual Representations: Instead of a single feature vector, each video clip is split into two halves and projected into dual features ($q^1, q^2$).
  2. Shuffle-Rank Pretext Task: To capture internal differences, sub-clips are shuffled, and the model is trained using a logistic ranking loss. This teaches the model that the first half of a clip should be more similar to its own augmented version than to the second half, providing a nuanced signal for temporal evolution.
  3. Temporal Coherent Contrast (TCC): To preserve the "big picture," TCC ensures video-level identity by averaging the similarities of the dual sub-clip features. This ensures that while sub-clips are recognized as distinct, they still cluster together relative to clips from other videos.
The DualVar framework architecture featuring Temporal Coherent Contrast and Shuffle-Rank tasks.

Experiments and Results

The DualVar framework was evaluated on industry-standard benchmarks including Kinetics-400, UCF101, and HMDB51. The results demonstrate that incorporating intra-variance consistently improves performance across various neural architectures, such as R3D, R(2+1)D, and S3D-G.

Action Recognition and Retrieval

When integrated with SimCLR and pretrained on Kinetics-400, the model achieved 82.0% accuracy on UCF101 and 51.2% on HMDB51, outperforming previous state-of-the-art methods like RSPNet. In video retrieval tasks, the model reached 46.7% Top-1 accuracy, a 5.6% improvement over previous benchmarks, indicating that the learned features are both discriminative and semantically rich.

Performance comparison showing the additive benefits of Shuffle-Rank (SR) and Temporal Coherent Contrast (TC) across different backbones.

Conclusion

DualVar highlights a critical oversight in current contrastive learning methods: the neglect of intra-video variance. By allowing for a controlled amount of internal difference through dual representations and ranking losses, the model learns a more complete view of video data. This approach not only improves accuracy on standard benchmarks but also enhances the model's ability to generalize to new, unseen actions, marking a significant step forward in self-supervised video understanding.

Summary updated

BibTeX
@inproceedings{zhang2021how,
  author = {Lin Zhang and Qi She and Zhengyang Shen and Changhu Wang},
  title = {{How Incomplete is Contrastive Learning? An Inter-intra Variant Dual Representation Method for Self-supervised Video Recognition}},
  booktitle = {BMVC},
  year = {2021},
}
Figure