Abstract
Unsupervised video representation learning typically relies on manually designed pretext tasks. This work introduces an AutoML approach that frames representation learning as a multi-modal, multi-task optimization problem. By employing an evolutionary algorithm to discover optimal weightings for various self-supervised losses and cross-modal distillation, the model learns robust features from 2 million uncurated YouTube videos. The resulting representations achieve state-of-the-art performance on action recognition benchmarks, significantly narrowing the gap between unsupervised and supervised pretraining.
Introduction
Self-supervised learning leverages "pretext tasks"—such as frame order prediction or colorization—to learn visual features from unlabeled data. Recent advances suggest that combining multiple tasks across modalities (e.g., RGB, audio, optical flow) improves representation quality. However, determining the relative importance of these tasks often requires exhaustive manual tuning. Furthermore, many "unsupervised" methods are evaluated on curated datasets like Kinetics, which contain inherent human-centric biases. This paper addresses these issues by automating loss function design and training on truly raw, uncurated video data.
Methodology
The proposed framework utilizes a multi-modal architecture processing RGB, optical flow, and audio streams.
Multi-Modal Distillation
To consolidate knowledge into a single RGB-based model, the authors employ cross-modal distillation. This process forces the internal activations of the RGB network to align with those of specialized audio and flow networks, effectively infusing multi-modal knowledge into the primary visual model.
Evolutionary Loss Optimization
The core innovation is the use of an evolutionary algorithm to optimize the weights ($\lambda$) for each pretext task and distillation loss. A population of weight combinations is maintained and evolved over thousands of rounds. Fitness is determined by training a network on unlabeled data and evaluating the cluster quality of its representations on a small subset of the HMDB dataset. The best-performing weights are mutated and carried over to subsequent generations.
Experimental Setup
The model was trained on a new dataset of 2 million random 10-second YouTube clips, representing a truly uncurated "in-the-wild" source. Evaluation was conducted on standard benchmarks including HMDB51, UCF101, and Kinetics to measure transferability to downstream action recognition tasks.
Key Insights from Evolution
The evolutionary search provided unique insights into the utility of common pretext tasks and the dynamics of multi-modal learning.
- Task Prioritization: Tasks such as audio-video alignment and cross-modality reconstruction (e.g., predicting flow from RGB) were assigned high weights. Conversely, popular tasks like frame shuffling and colorization were found to be less effective for action recognition and received low weights.
- Distillation Hierarchy: The algorithm discovered that low-level features (grayscale) are best distilled early in the network, whereas more abstract semantic features (audio and optical flow) should be distilled in deeper layers.
Results
The evolved model sets a new standard for unsupervised video representation:
- Benchmark Performance: It achieved 66.2% accuracy on HMDB51 and 92.4% on UCF101, significantly outperforming previous state-of-the-art self-supervised methods like AVTS (61.6% / 89.0%).
- Transfer Efficiency: The unsupervised model outperformed ImageNet-pretrained baselines (49.8% on HMDB51) and neared the performance of Kinetics-supervised pretraining (74.3%).
- Data Efficiency: Using only 10% of labeled Kinetics data for fine-tuning yielded 58.1% accuracy, demonstrating that the learned features are highly robust even with limited supervision.
Conclusion and Limitations
This work demonstrates that powerful video representations can be learned from massive, uncurated data by automatically optimizing the learning objective. While the evolutionary approach is computationally intensive, requiring significant resources typical of large-scale research labs, it provides a blueprint for moving beyond hand-crafted pretext tasks in self-supervised learning. Future work may explore more efficient search strategies to make this automated design process more accessible.
Summary updated