DABS: a Domain-Agnostic Benchmark for Self-Supervised Learning

Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, Noah D. Goodman

TL;DR

Current self-supervised learning (SSL) techniques rely heavily on hand-crafted, domain-specific inductive biases—such as color jitter in vision or masked language modeling in NLP—limiting their utility for less-studied data modalities. To bridge this gap, the authors introduce DABS, a multi-domain benchmark that evaluates SSL algorithms across seven diverse data domains using fixed model architectures and objectives. Evaluating two new modality-agnostic baseline algorithms (e-Mix and ShED) reveals that while generic pretraining improves performance over random initialization, substantial room remains before SSL can serve as a true plug-and-play solution.

What Problem Does This Paper Address?

While SSL has revolutionized NLP, computer vision, and speech processing, modern methods remain siloed. Techniques engineered for one domain cannot easily transfer to another: color jitter fails on monochromatic chest X-rays, and masked token prediction does not cleanly translate to unsegmented raw sensor feeds or continuous audio spectrograms.

Developing custom SSL pipelines for the "long tail" of domains (such as biomedical datasets, multi-channel wearables, and multi-modal setups) requires extensive trial-and-error by ML specialists. DABS addresses the lack of a unified evaluation platform to test whether an SSL algorithm can learn useful representations across diverse data formats without human-designed, domain-specific adaptations.

Methods

DABS establishes strict standardization rules to measure domain agnosticism:

  • Unified Backbone: A standardized 12-layer Transformer (256 hidden size, 8 heads) is kept identical across all experiments.
  • Minimal Modality Adapters: Input differences are handled strictly at the boundary using minimal embedding layers: 2D patch projection for images and spectrograms, 1D patch projection for multi-channel sensor series, and standard token lookups for text.
  • Strict Protocol: Domain-specific data augmentations (e.g., random resized cropping) are prohibited. Downstream evaluation is conducted via a frozen linear classifier trained on mean-pooled representation vectors.

The paper introduces two domain-agnostic SSL baseline objectives that operate directly in embedding space:

  1. e-Mix: A contrastive objective extending mixup to embedding space. Vectors from different examples are linearly interpolated ($\tilde{x}_i = \lambda x_i + (1-\lambda)x_{\pi(i)}$), and the model is trained with InfoNCE-style contrastive loss to predict both sources proportionally.
  2. ShED (Shuffled Embedding Detection): A continuous adaptation of ELECTRA. Instead of masking or replacing tokens with dictionary items, 15% of embedding positions are swapped via cyclic derangements. The network solves a binary classification task predicting which embedding vectors were shuffled.

Datasets

DABS spans seven domains, pairing an unlabeled pretraining dataset with one or more downstream transfer tasks (covering 25+ datasets in total):

  • Natural Images: Pretrained on ImageNet; evaluated on CIFAR-10, DTD Textures, FGVC-Aircraft, CUB-200 Birds, GTSRB, and VGG Flowers.
  • Speech Recordings: Pretrained on LibriSpeech (log-mel spectrograms); evaluated on VoxCeleb, LibriSpeech Speaker ID, Fluent Speech Commands, Google Speech Commands, and AudioMNIST.
  • Monolingual English Text: Pretrained on WikiText-103; evaluated on the GLUE benchmark.
  • Multilingual Text: Pretrained on an interleaved 7-language split of mC4 (EN, ES, FR, DE, ZH, JA, KO); evaluated on PAWS-X.
  • Medical Imaging: Pretrained on CheXpert chest radiographs; evaluated on CheXpert's five competition pathologies and ChestX-ray8.
  • Multi-channel Sensor Data: Pretrained on 52-channel PAMAP2 wearable telemetry; evaluated on physical activity recognition.
  • Images with Descriptions: Pretrained on paired MS COCO image-text samples; evaluated on MS COCO retrieval and binary Visual Question Answering (VQA).

Key Insights and Ideas

  • Inherited Inductive Biases Matter: Baseline success closely mirrored the algorithm’s family origin. Contrastive e-Mix achieved top performance on continuous, vision-like domains (e.g., 27.9% average accuracy on natural images vs. 20.9% for ShED), whereas prediction-based ShED excelled on structured/sequence-heavy data (e.g., 48.4 text score vs. 44.1 for e-Mix, and 88.7% sensor accuracy vs. 79.5%).
  • Multimodal Failure Mode: Self-supervised pretraining on image-text pairs actually degraded performance on downstream VQA compared to an untrained, randomly initialized model (57.5% down to 48.9% with e-Mix and 54.3% with ShED), exposing the difficulty of cross-modal alignment without handcrafted multimodal objectives.
  • Significant Headroom: Although generic SSL beat random baselines across unimodal tasks, accuracy gains were modest compared to specialized, domain-specific methods, confirming that domain-agnostic representation learning remains an open challenge.

Key Results

Metrics reflect average linear readout performance across each domain's transfer suite:

Domain Untrained Baseline e-Mix (Contrastive) ShED (Prediction)
Natural Images (Avg. Acc %) 10.1% 27.9% 20.9%
Speech (Avg. Acc %) 24.9% 41.8% 36.5%
English Text (GLUE Avg.) 42.3 44.1 48.4
Wearable Sensors (PAMAP2 Acc %) 69.8% 79.5% 88.7%
Chest X-Rays (Avg. AUROC %) 68.1% 72.4% 74.5%
Images & Text (VQA Acc %) 57.5% 48.9% 54.3%

Limitations and Future Work

  • Domain Coverage: Although diverse, seven domains represent a fraction of scientific data types; the authors intend to add more modalities over time.
  • Exclusion of Hybrid Approaches: The benchmark strictly penalizes domain-specific augmentations, meaning it does not evaluate how general-purpose objectives perform when combined with domain-specific priors.
  • Architecture Constraints: Standard Transformer sequence processing with patch projection imposes minimal locality assumptions. Exploring architectures like Perceiver that entirely forgo positional patch assumptions remains an open direction.

Summary updated

Abstract

Self-supervised learning algorithms, including BERT and SimCLR, have enabled significant strides in fields like natural language processing, computer vision, and speech processing. However, these algorithms are domain-specific, meaning that new self-supervised learning algorithms must be developed for each new setting, including myriad healthcare, scientific, and multimodal domains. To catalyze progress toward domain-agnostic methods, we introduce DABS: a Domain-Agnostic Benchmark for Self-supervised learning. To perform well on DABS, an algorithm is evaluated on seven diverse domains: natural images, multichannel sensor data, English text, speech recordings, multilingual text, chest x-rays, and images with text descriptions. Each domain contains an unlabeled dataset for pretraining; the model is then is scored based on its downstream performance on a set of labeled tasks in the domain. We also present e-Mix and ShED: two baseline domain-agnostic algorithms; their relatively modest performance demonstrates that significant progress is needed before self-supervised learning is an out-of-the-box solution for arbitrary domains. Code for benchmark datasets and baseline algorithms is available at https://github.com/alextamkin/dabs.
BibTeX
@inproceedings{tamkin2021dabs,
  author = {Alex Tamkin and Vincent Liu and Rongfei Lu and Daniel Fein and Colin Schultz and Noah D. Goodman},
  title = {{DABS: a Domain-Agnostic Benchmark for Self-Supervised Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=u4-6I7y9P8},
}

Loading related content...

Figure