Abstract
Vi2CLR (Video/Image for Visual Contrastive Learning of Representation) is a self-supervised framework designed to bridge the gap between static and dynamic representation learning. By jointly training 2D and 3D convolutional neural networks on unlabeled video data, Vi2CLR leverages both spatial and temporal cues. The framework combines instance-level contrastive learning with a novel cluster-centroid loss ($L_{CenterNCE}$) using the parameter-free FINCH algorithm. Experimental results demonstrate that Vi2CLR achieves state-of-the-art performance in both image classification (74.6% top-1 on ImageNet) and video action recognition (88.9% on UCF101), proving the effectiveness of cross-modal synergy in self-supervised learning.
Introduction
In computer vision, representation learning aims to teach models to understand the visual world without extensive manual labeling. Current self-supervised learning (SSL) methods typically specialize in either static images (spatial cues) or videos (temporal dynamics). However, human perception naturally integrates both modalities. Existing SSL approaches often miss this synergy; video-only models may overlook fine-grained object details, while image-only models remain blind to motion. Vi2CLR addresses this by proposing a unified pipeline that learns from both modalities simultaneously using a single source of unlabeled video clips.
Methodology
The Vi2CLR architecture consists of a dual-stream pipeline that processes video clips in two distinct ways:
- 3D Stream: A 3D ConvNet (S3D) processes the entire video clip to capture temporal motion.
- 2D Stream: A 2D ConvNet (ResNet-50) processes the middle frame of the same clip to capture high-resolution static features.
The model is optimized using three contrastive losses:
- $L_{VideoNCE}$ and $L_{ImgNCE}$: Standard instance-discrimination losses that ensure augmented versions of the same sample remain close in feature space while pushing different samples apart.
- $L_{CenterNCE}$: This novel loss concatenates features from both streams and performs clustering using the FINCH algorithm. It pulls the joint representation of an instance toward its assigned cluster centroid, providing a high-level semantic signal that groups similar concepts (e.g., different clips of "cycling") together.
By alternating between feature learning and clustering, the model moves from simple instance awareness to category awareness without human supervision.
Experiments and Results
The model was pre-trained on the Kinetics-400 dataset (~250,000 unlabeled clips). Evaluation spanned both video and image domains.
Video Evaluation
On UCF101, Vi2CLR achieved 88.9% fine-tuned accuracy, surpassing previous leaders like CoCLR and CVRL. In video retrieval tasks, the model showed a significant jump in Recall@1 (R@1) on UCF101 to 55.4%, indicating highly discriminative learned features.
Image Evaluation
The 2D encoder was evaluated on ImageNet (ILSVRC-2012) and Pascal VOC’07. Despite being trained on video frames, the model reached 74.6% top-1 accuracy on ImageNet using a linear classifier on frozen features.
As shown in the results, Vi2CLR outperforms popular image-only SSL methods such as SimCLR (69.3%) and SwAV (74.2%). This success highlights the benefits of cross-modal training, where the 2D network learns features more relevant to real-world actions through its association with the 3D stream.
Key Insights
- Anchor Frames: Using the middle frame as the 2D input serves as a reliable anchor for the static features of an action.
- Clustering as Pseudo-Supervision: Clustering allows the model to discover natural partitions in the data. Using FINCH instead of K-Means avoids the need to pre-specify the number of clusters, typically discovering 10K–15K high-purity clusters.
- Synergy: Training both networks together improves spatial precision in the 3D stream and action-relevance in the 2D stream.
Conclusion
Vi2CLR demonstrates that joint training on video and image modalities significantly enhances representation learning. While the approach is computationally intensive, requiring substantial GPU resources, it sets a new benchmark for versatile SSL models. Future work may explore scaling this method to larger "in-the-wild" datasets and investigating longer-term temporal dependencies.
Summary updated