Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language

Alexei Baevski, Arun Babu, Wei-Ning Hsu, Michael Auli

Abstract

Current self-supervised learning algorithms are often modality-specific and require large amounts of computational resources. To address these issues, we increase the training efficiency of data2vec, a learning objective that generalizes across several modalities. We do not encode masked tokens, use a fast convolutional decoder and amortize the effort to build teacher representations. data2vec 2.0 benefits from the rich contextualized target representations introduced in data2vec which enable a fast self-supervised learner. Experiments on ImageNet-1K image classification show that data2vec 2.0 matches the accuracy of Masked Autoencoders in 16.4x lower pre-training time, on Librispeech speech recognition it performs as well as wav2vec 2.0 in 10.6x less time, and on GLUE natural language understanding it matches a retrained RoBERTa model in half the time. Trading some speed for accuracy results in ImageNet-1K top-1 accuracy of 86.8% with a ViT-L model trained for 150 epochs.

Introduction

Self-supervised learning (SSL) has revolutionized artificial intelligence by enabling models to learn from vast quantities of unlabeled data. Despite this success, two significant challenges persist: the prevalence of modality-specific "silos" (e.g., MAE for vision, wav2vec for speech, BERT for text) and the immense computational cost associated with training these models.

While the original data2vec framework established a unified learning objective across modalities, its efficiency was limited. The teacher model processed every sample, and the student model often handled redundant information. Data2vec 2.0 addresses these bottlenecks, maintaining a unified objective while achieving training speeds orders of magnitude faster than previous state-of-the-art methods.

Methods: The Efficiency Engine

Data2vec 2.0 introduces several architectural and algorithmic optimizations designed to minimize computational overhead without sacrificing representation quality.

Overview of the data2vec 2.0 framework across language, speech, and vision modalities, illustrating the multi-mask training strategy.

Asymmetric Encoder/Decoder

Following the success of Masked Autoencoders (MAE), the student model in data2vec 2.0 only processes unmasked input tokens. This significantly reduces the workload for the Transformer encoder. To reconstruct the target representations, the model employs a lightweight, multi-layer convolutional decoder, which is substantially faster than a full Transformer-based decoder.

Multi-mask Training

The core innovation of data2vec 2.0 is the amortization of teacher representation costs. In traditional SSL, the teacher processes each sample once for every student update. In data2vec 2.0, the teacher processes an unmasked sample a single time to generate targets. The student then attempts to predict these same targets for $M$ different masked versions of that sample simultaneously. This approach allows for multiple student updates per teacher pass, dramatically reducing wall-clock training time.

Inverse Block Masking

To ensure a challenging and informative learning task, the model uses inverse block masking. Rather than masking random individual patches, it preserves contiguous blocks of data. This forces the student to rely on global context rather than simple local interpolation to predict the missing parts.

Experimental Setup and Datasets

The model's performance and efficiency were evaluated across three distinct domains:

  • Vision: Pre-trained on ImageNet-1K for image classification.
  • Speech: Pre-trained on Librispeech (960 hours) and Libri-light (60k hours) for speech recognition.
  • NLP: Pre-trained on the Books Corpus and English Wikipedia, evaluated on the GLUE benchmark.

Results and Insights

The efficiency gains of data2vec 2.0 are substantial across all modalities when compared to established baselines at equivalent accuracy levels:

  • Computer Vision: Matches MAE accuracy while being 16.4x faster in pre-training.
  • Speech Processing: Matches wav2vec 2.0 performance while being 10.6x faster.
  • Natural Language Processing: Matches a retrained RoBERTa model in half the time.

A key takeaway is that contextualized targets—predicting the internal representations of a teacher model rather than raw pixels or discrete tokens—provide a superior learning signal. These targets already incorporate information about the relationship between a specific part of the data and the whole, making the learning task more robust.

Conclusion and Future Work

Data2vec 2.0 demonstrates that a single, efficient objective—predicting contextualized latent representations—can dominate modality-specific approaches. While the learning objective is unified, the initial feature encoders remain modality-specific. Future research may focus on developing truly universal encoders and extending this efficient framework to additional modalities such as video or sensor data.

Summary updated

Abstract

Current self-supervised learning algorithms are often modality-specific and require large amounts of computational resources. To address these issues, we increase the training efficiency of data2vec, a learning objective that generalizes across several modalities. We do not encode masked tokens, use a fast convolutional decoder and amortize the effort to build teacher representations. data2vec 2.0 benefits from the rich contextualized target representations introduced in data2vec which enable a fast self-supervised learner. Experiments on ImageNet-1K image classification show that data2vec 2.0 matches the accuracy of Masked Autoencoders in 16.4x lower pre-training time, on Librispeech speech recognition it performs as well as wav2vec 2.0 in 10.6x less time, and on GLUE natural language understanding it matches a retrained RoBERTa model in half the time. Trading some speed for accuracy results in ImageNet-1K top-1 accuracy of 86.8\% with a ViT-L model trained for 150 epochs.
BibTeX
@inproceedings{baevski2023efficient,
  author = {Alexei Baevski and Arun Babu and Wei-Ning Hsu and Michael Auli},
  title = {{Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=Jc5QwxfyyQ},
}
Figure