Vi2CLR: Video and Image for Visual Contrastive Learning of Representation

Ali Diba, Vivek Sharma, Reza Safdari, Dariush Lotfi, M. Saquib Sarfraz, Rainer Stiefelhagen, Luc Van Gool

Abstract

Vi2CLR (Video/Image for Visual Contrastive Learning of Representation) is a self-supervised framework designed to bridge the gap between static and dynamic representation learning. By jointly training 2D and 3D convolutional neural networks on unlabeled video data, Vi2CLR leverages both spatial and temporal cues. The framework combines instance-level contrastive learning with a novel cluster-centroid loss ($L_{CenterNCE}$) using the parameter-free FINCH algorithm. Experimental results demonstrate that Vi2CLR achieves state-of-the-art performance in both image classification (74.6% top-1 on ImageNet) and video action recognition (88.9% on UCF101), proving the effectiveness of cross-modal synergy in self-supervised learning.

Introduction

In computer vision, representation learning aims to teach models to understand the visual world without extensive manual labeling. Current self-supervised learning (SSL) methods typically specialize in either static images (spatial cues) or videos (temporal dynamics). However, human perception naturally integrates both modalities. Existing SSL approaches often miss this synergy; video-only models may overlook fine-grained object details, while image-only models remain blind to motion. Vi2CLR addresses this by proposing a unified pipeline that learns from both modalities simultaneously using a single source of unlabeled video clips.

Methodology

The Vi2CLR architecture consists of a dual-stream pipeline that processes video clips in two distinct ways:

  1. 3D Stream: A 3D ConvNet (S3D) processes the entire video clip to capture temporal motion.
  2. 2D Stream: A 2D ConvNet (ResNet-50) processes the middle frame of the same clip to capture high-resolution static features.
The Vi2CLR architecture showing dual-stream encoders and the triple-loss optimization framework.

The model is optimized using three contrastive losses:

  • $L_{VideoNCE}$ and $L_{ImgNCE}$: Standard instance-discrimination losses that ensure augmented versions of the same sample remain close in feature space while pushing different samples apart.
  • $L_{CenterNCE}$: This novel loss concatenates features from both streams and performs clustering using the FINCH algorithm. It pulls the joint representation of an instance toward its assigned cluster centroid, providing a high-level semantic signal that groups similar concepts (e.g., different clips of "cycling") together.

By alternating between feature learning and clustering, the model moves from simple instance awareness to category awareness without human supervision.

Experiments and Results

The model was pre-trained on the Kinetics-400 dataset (~250,000 unlabeled clips). Evaluation spanned both video and image domains.

Video Evaluation

On UCF101, Vi2CLR achieved 88.9% fine-tuned accuracy, surpassing previous leaders like CoCLR and CVRL. In video retrieval tasks, the model showed a significant jump in Recall@1 (R@1) on UCF101 to 55.4%, indicating highly discriminative learned features.

Image Evaluation

The 2D encoder was evaluated on ImageNet (ILSVRC-2012) and Pascal VOC’07. Despite being trained on video frames, the model reached 74.6% top-1 accuracy on ImageNet using a linear classifier on frozen features.

Linear classification results on ImageNet and Pascal VOC07, comparing Vi2CLR with other self-supervised methods.

As shown in the results, Vi2CLR outperforms popular image-only SSL methods such as SimCLR (69.3%) and SwAV (74.2%). This success highlights the benefits of cross-modal training, where the 2D network learns features more relevant to real-world actions through its association with the 3D stream.

Key Insights

  • Anchor Frames: Using the middle frame as the 2D input serves as a reliable anchor for the static features of an action.
  • Clustering as Pseudo-Supervision: Clustering allows the model to discover natural partitions in the data. Using FINCH instead of K-Means avoids the need to pre-specify the number of clusters, typically discovering 10K–15K high-purity clusters.
  • Synergy: Training both networks together improves spatial precision in the 3D stream and action-relevance in the 2D stream.

Conclusion

Vi2CLR demonstrates that joint training on video and image modalities significantly enhances representation learning. While the approach is computationally intensive, requiring substantial GPU resources, it sets a new benchmark for versatile SSL models. Future work may explore scaling this method to larger "in-the-wild" datasets and investigating longer-term temporal dependencies.

Summary updated

Abstract

In this paper, we introduce a novel self-supervised visual representation learning method which understands both images and videos in a joint learning fashion. The proposed neural network architecture and objectives are designed to obtain two different Convolutional Neural Networks for solving visual recognition tasks in the domain of videos and images. Our method called Video/Image for Visual Contrastive Learning of Representation(Vi <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> CLR) uses unlabeled videos to exploit dynamic and static visual cues for self-supervised and instances similarity/dissimilarity learning. Vi <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> CLR optimization pipeline consists of visual clustering part and representation learning based on groups of similar positive instances within a cluster and negative ones from other clusters and learning visual clusters and their distances. We show how a joint self-supervised visual clustering and instance similarity learning with 2D (image) and 3D (video) CovNet encoders yields such robust and near to supervised learning performance.We extensively evaluate the method on downstream tasks like large scale action recognition, image and object classification on datasets like Kinetics, ImageNet, Pascal VOC’07 and UCF101 and achieve outstanding results compared to state-of-the-art self-supervised methods.
BibTeX
@inproceedings{diba2021vi,
  author = {Ali Diba and Vivek Sharma and Reza Safdari and Dariush Lotfi and M. Saquib Sarfraz and Rainer Stiefelhagen and Luc Van Gool},
  title = {{Vi2CLR: Video and Image for Visual Contrastive Learning of Representation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00153},
}
Figure