Labelling unlabelled videos from scratch with multi-modal self-supervision

Yuki Markus Asano, Mandela Patrick, Christian Rupprecht, Andrea Vedaldi

Abstract

A large part of the current success of deep learning lies in the effectiveness of data -- more precisely: labelled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningful (pseudo-) labels for unlabelled datasets without supervision, this development is missing for the video domain where learning feature representations is the current focus. In this work, we a) show that unsupervised labelling of a video dataset does not come for free from strong feature encoders and b) propose a novel clustering method that allows pseudo-labelling of a video dataset without any human annotations, by leveraging the natural correspondence between the audio and visual modalities. An extensive analysis shows that the resulting clusters have high semantic overlap to ground truth human labels. We further introduce the first benchmarking results on unsupervised labelling of common video datasets Kinetics, Kinetics-Sound, VGG-Sound and AVE.

Introduction

Deep learning's reliance on massive labelled datasets is a significant bottleneck, particularly for video content. Annotating video is labor-intensive, requiring humans to watch clips, identify temporal boundaries, and often process audio. While "self-labelling" techniques have matured for static images, the video domain has primarily focused on representation learning rather than explicit categorization.

Researchers from the University of Oxford’s Visual Geometry Group (VGG) and Facebook AI Research (FAIR) address this gap with SeLaVi (Self-Labelling Videos). This framework exploits the inherent correspondence between sight and sound to automatically cluster and label video data from scratch, matching human-defined categories without manual intervention.

The SeLaVi Framework

SeLaVi builds upon image-based self-labelling methods by introducing innovations tailored for multi-modal video data. The core concept is to treat audio and visual streams as different "augmentations" of the same semantic concept.

Overview of the SeLaVi framework showing multi-modal encoding and Sinkhorn clustering.

The method involves several key components:

  1. Multi-modal Splicing: The model is trained to ensure that labels predicted from the audio stream match those from the video stream.
  2. Optimal Transport with Skewed Priors: Real-world data often follows a long-tailed distribution. SeLaVi adapts the Sinkhorn-Knopp algorithm to handle these skewed priors, moving away from the assumption of equally likely categories.
  3. Initialization and Alignment: To synchronize the separate audio and video networks from the start, a permutation matrix is used to align the output layers, ensuring both modalities "speak the same language" regarding clusters.
  4. Decorrelated Heads: Multiple clustering functions are learned in parallel and forced to be distinct, allowing the model to capture various semantic aspects of the data (e.g., grouping by action vs. environment).

Experimental Setup

The framework was evaluated on four diverse benchmarks:

  • VGG-Sound: 200k videos across 309 categories.
  • AVE (Audio-Visual Event): Focused on event localization.
  • Kinetics-400: The standard for human action recognition.
  • Kinetics-Sound: A subset of Kinetics with distinct audio-visual actions.

Results and Discussion

A key insight from the study is that high-quality feature representations do not automatically lead to high-quality clusters; explicit optimization for clustering is necessary. SeLaVi demonstrates that audio is a powerful supervisor, allowing the model to learn abstract concepts by grouping visually distinct scenes that share common acoustic signatures.

Comparison of NMI performance across different compression levels.

SeLaVi achieved state-of-the-art results in unsupervised labelling:

  • VGG-Sound: Reached a 56.7% NMI, far exceeding previous methods like XDC (18.1%).
  • AVE: Correctly grouped 55% of videos into ground-truth categories without any labels.
  • Robustness: As shown in the performance analysis, SeLaVi maintains superior Normalized Mutual Information (NMI) even as video compression increases, outperforming both self-supervised and supervised baselines.

Furthermore, the learned features proved highly effective for downstream tasks. On the UCF-101 video retrieval benchmark, SeLaVi achieved a 52.0% Recall@1, doubling the performance of prior self-supervised methods.

Conclusion and Future Work

While SeLaVi excels in audio-rich datasets, it faces challenges in datasets like Kinetics-400 where labels are purely visual and audio is often irrelevant noise. Future iterations could incorporate a third modality, such as text from automated speech recognition (ASR), to better distinguish complex human actions. The framework also holds promise for "few-label" applications, such as harmful content detection, where clustering can significantly amplify the impact of limited human annotations.

Summary updated

Abstract

A large part of the current success of deep learning lies in the effectiveness of data -- more precisely: labelled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningful (pseudo-) labels for unlabelled datasets without supervision, this development is missing for the video domain where learning feature representations is the current focus. In this work, we a) show that unsupervised labelling of a video dataset does not come for free from strong feature encoders and b) propose a novel clustering method that allows pseudo-labelling of a video dataset without any human annotations, by leveraging the natural correspondence between the audio and visual modalities. An extensive analysis shows that the resulting clusters have high semantic overlap to ground truth human labels. We further introduce the first benchmarking results on unsupervised labelling of common video datasets Kinetics, Kinetics-Sound, VGG-Sound and AVE.
BibTeX
@inproceedings{asano2020labelling,
  author = {Yuki Markus Asano and Mandela Patrick and Christian Rupprecht and Andrea Vedaldi},
  title = {{Labelling unlabelled videos from scratch with multi-modal self-supervision}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=UThWpQjhbfF},
}
Figure