Evolving Losses for Unlabeled Video Representation Learning

A. J. Piergiovanni, Anelia Angelova, Michael S. RyooarXiv 2019 · OpenReview

Abstract

Unsupervised video representation learning typically relies on manually designed pretext tasks. This work introduces an AutoML approach that frames representation learning as a multi-modal, multi-task optimization problem. By employing an evolutionary algorithm to discover optimal weightings for various self-supervised losses and cross-modal distillation, the model learns robust features from 2 million uncurated YouTube videos. The resulting representations achieve state-of-the-art performance on action recognition benchmarks, significantly narrowing the gap between unsupervised and supervised pretraining.

Introduction

Self-supervised learning leverages "pretext tasks"—such as frame order prediction or colorization—to learn visual features from unlabeled data. Recent advances suggest that combining multiple tasks across modalities (e.g., RGB, audio, optical flow) improves representation quality. However, determining the relative importance of these tasks often requires exhaustive manual tuning. Furthermore, many "unsupervised" methods are evaluated on curated datasets like Kinetics, which contain inherent human-centric biases. This paper addresses these issues by automating loss function design and training on truly raw, uncurated video data.

Methodology

The proposed framework utilizes a multi-modal architecture processing RGB, optical flow, and audio streams.

The multi-modal, multi-task framework incorporates distillation losses and task-specific weights across different data streams.

Multi-Modal Distillation

To consolidate knowledge into a single RGB-based model, the authors employ cross-modal distillation. This process forces the internal activations of the RGB network to align with those of specialized audio and flow networks, effectively infusing multi-modal knowledge into the primary visual model.

Evolutionary Loss Optimization

The core innovation is the use of an evolutionary algorithm to optimize the weights ($\lambda$) for each pretext task and distillation loss. A population of weight combinations is maintained and evolved over thousands of rounds. Fitness is determined by training a network on unlabeled data and evaluating the cluster quality of its representations on a small subset of the HMDB dataset. The best-performing weights are mutated and carried over to subsequent generations.

Experimental Setup

The model was trained on a new dataset of 2 million random 10-second YouTube clips, representing a truly uncurated "in-the-wild" source. Evaluation was conducted on standard benchmarks including HMDB51, UCF101, and Kinetics to measure transferability to downstream action recognition tasks.

Key Insights from Evolution

The evolutionary search provided unique insights into the utility of common pretext tasks and the dynamics of multi-modal learning.

The evolution of task weights over 2000 rounds reveals that tasks like audio-video alignment are prioritized over traditional tasks like frame shuffling.
  • Task Prioritization: Tasks such as audio-video alignment and cross-modality reconstruction (e.g., predicting flow from RGB) were assigned high weights. Conversely, popular tasks like frame shuffling and colorization were found to be less effective for action recognition and received low weights.
  • Distillation Hierarchy: The algorithm discovered that low-level features (grayscale) are best distilled early in the network, whereas more abstract semantic features (audio and optical flow) should be distilled in deeper layers.

Results

The evolved model sets a new standard for unsupervised video representation:

  • Benchmark Performance: It achieved 66.2% accuracy on HMDB51 and 92.4% on UCF101, significantly outperforming previous state-of-the-art self-supervised methods like AVTS (61.6% / 89.0%).
  • Transfer Efficiency: The unsupervised model outperformed ImageNet-pretrained baselines (49.8% on HMDB51) and neared the performance of Kinetics-supervised pretraining (74.3%).
  • Data Efficiency: Using only 10% of labeled Kinetics data for fine-tuning yielded 58.1% accuracy, demonstrating that the learned features are highly robust even with limited supervision.

Conclusion and Limitations

This work demonstrates that powerful video representations can be learned from massive, uncurated data by automatically optimizing the learning objective. While the evolutionary approach is computationally intensive, requiring significant resources typical of large-scale research labs, it provides a blueprint for moving beyond hand-crafted pretext tasks in self-supervised learning. Future work may explore more efficient search strategies to make this automated design process more accessible.

Summary updated

Abstract

We present a new method to learn video representations from unlabeled data. Given large-scale unlabeled video data, the objective is to benefit from such data by learning a generic and transferable representation space that can be directly used for a new task such as zero/few-shot learning. We formulate our unsupervised representation learning as a multi-modal, multi-task learning problem, where the representations are also shared across different modalities via distillation. Further, we also introduce the concept of finding a better loss function to train such multi-task multi-modal representation space using an evolutionary algorithm; our method automatically searches over different combinations of loss functions capturing multiple (self-supervised) tasks and modalities. Our formulation allows for the distillation of audio, optical flow and temporal information into a single, RGB-based convolutional neural network. We also compare the effects of using additional unlabeled video data and evaluate our representation learning on standard public video datasets.
BibTeX
@misc{piergiovanni2019evolving,
  author = {A. J. Piergiovanni and Anelia Angelova and Michael S. Ryoo},
  title = {{Evolving Losses for Unlabeled Video Representation Learning}},
  year = {2019},
  eprint = {1906.03248},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.03248},
}
Figure