DABS: A Domain-Agnostic Benchmark for Self-Supervised Learning

Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, Noah D. Goodman

Abstract

Self-supervised learning algorithms, including BERT and SimCLR, have enabled significant strides in fields like natural language processing, computer vision, and speech processing. However, these algorithms are domain-specific, meaning that new self-supervised learning algorithms must be developed for each new setting, including myriad healthcare, scientific, and multimodal domains. To catalyze progress toward domain-agnostic methods, we introduce DABS: a Domain-Agnostic Benchmark for Self-supervised learning. To perform well on DABS, an algorithm is evaluated on seven diverse domains: natural images, multichannel sensor data, English text, speech recordings, multilingual text, chest x-rays, and images with text descriptions. Each domain contains an unlabeled dataset for pretraining; the model is then is scored based on its downstream performance on a set of labeled tasks in the domain. We also present e-Mix and ShED: two baseline domain-agnostic algorithms; their relatively modest performance demonstrates that significant progress is needed before self-supervised learning is an out-of-the-box solution for arbitrary domains.

Introduction

Self-supervised learning (SSL) has revolutionized artificial intelligence, yielding state-of-the-art models like BERT for natural language processing and SimCLR for computer vision. However, these successes are largely "hand-crafted," relying on domain-specific expertise. For instance, SimCLR requires specific image augmentations like color jittering, while BERT necessitates a discrete tokenizer. This dependency creates a "long tail" problem: niche datasets—such as multichannel sensor data from industrial settings or rare medical scans—often lack the specialized ML expertise or "augmentation recipes" required to train effective SSL models.

To address this, we argue that SSL must become domain-agnostic. An ideal algorithm should ingest raw data from any modality and learn useful representations without human intervention. We introduce the Domain-Agnostic Benchmark for Self-supervised learning (DABS) to catalyze research in this direction.

The DABS Framework

DABS evaluates models across seven diverse domains using a standardized pipeline where the model architecture, pretraining objective, and transfer method are held constant.

The DABS framework evaluates domain-agnostic SSL across seven diverse data modalities.

The benchmark covers the following domains:

  • Natural Images: ImageNet pretraining with transfer to tasks like CIFAR-10.
  • English Text: WikiText-103 pretraining evaluated on the GLUE benchmark.
  • Speech: LibriSpeech recordings for tasks like speaker identification.
  • Multilingual Text: A subset of the mC4 dataset covering seven languages.
  • Medical Imaging: Chest x-rays from the CheXpert dataset.
  • Sensor Data: Multichannel wearable sensor data from PAMAP2.
  • Images with Text: MS COCO captioned images for multimodal evaluation.

Methods and Baselines

To handle diverse inputs, DABS uses a standard 12-layer Transformer architecture. Raw data is converted into a sequence of vectors via "embedding modules" (e.g., linear patches for images or lookup tables for text). We introduce two baseline domain-agnostic algorithms:

  1. e-Mix (Contrastive): This generalizes "mixup" data augmentation by performing it in the embedding space rather than on raw pixels. The model is trained to recognize the original components of the mixed embedding using a contrastive loss.
  2. ShED (Predictive): Short for Shuffled Embedding Detection, this is a domain-agnostic version of the ELECTRA objective. It shuffles a subset of input embeddings and trains the model to identify which positions were swapped.

Key Insights and Results

Our experiments reveal several critical findings:

  • "Origin" Bias: Algorithms tend to retain the strengths of their fields of origin. e-Mix, being contrastive (common in CV), performed best on images, while ShED, being predictive (common in NLP), excelled on text and sensor data.
  • Embedding-Space Augmentation: Moving augmentations to the embedding level successfully bypasses the need for domain-specific tricks like cropping or masking.
  • The Multimodal Challenge: Surprisingly, baseline SSL methods occasionally hurt performance in the image-text domain compared to no pretraining. This suggests that inter-modality relationships are significantly harder to learn than intra-modality ones in an agnostic pipeline.

While results show that domain-agnostic SSL is viable—with ShED improving sensor data performance from 69.8% to 88.7%—a significant gap remains between these baselines and domain-specific state-of-the-art models.

Conclusion and Future Work

DABS provides a rigorous testing ground for universal AI. Current limitations include uneven gains across domains and architectural constraints of the Transformer. Future work will explore more flexible architectures, like the Perceiver, and expand the benchmark to include more "long-tail" scientific and healthcare domains. DABS serves as the "gym" for universal AI; the goal now is to develop the "athlete" capable of competing across all events.

Summary updated

Abstract

Self-supervised learning algorithms, including BERT and SimCLR, have enabled significant strides in fields like natural language processing, computer vision, and speech processing. However, these algorithms are domain-specific, meaning that new self-supervised learning algorithms must be developed for each new setting, including myriad healthcare, scientific, and multimodal domains. To catalyze progress toward domain-agnostic methods, we introduce DABS: a Domain-Agnostic Benchmark for Self-supervised learning. To perform well on DABS, an algorithm is evaluated on seven diverse domains: natural images, multichannel sensor data, English text, speech recordings, multilingual text, chest x-rays, and images with text descriptions. Each domain contains an unlabeled dataset for pretraining; the model is then is scored based on its downstream performance on a set of labeled tasks in the domain. We also present e-Mix and ShED: two baseline domain-agnostic algorithms; their relatively modest performance demonstrates that significant progress is needed before self-supervised learning is an out-of-the-box solution for arbitrary domains. Code for benchmark datasets and baseline algorithms is available at https://github.com/alextamkin/dabs.
BibTeX
@inproceedings{tamkin2021dabs,
  author = {Alex Tamkin and Vincent Liu and Rongfei Lu and Daniel Fein and Colin Schultz and Noah D. Goodman},
  title = {{DABS: A Domain-Agnostic Benchmark for Self-Supervised Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=u4-6I7y9P8},
}
Figure