TL;DR
Current self-supervised learning (SSL) techniques rely heavily on hand-crafted, domain-specific inductive biases—such as color jitter in vision or masked language modeling in NLP—limiting their utility for less-studied data modalities. To bridge this gap, the authors introduce DABS, a multi-domain benchmark that evaluates SSL algorithms across seven diverse data domains using fixed model architectures and objectives. Evaluating two new modality-agnostic baseline algorithms (e-Mix and ShED) reveals that while generic pretraining improves performance over random initialization, substantial room remains before SSL can serve as a true plug-and-play solution.
What Problem Does This Paper Address?
While SSL has revolutionized NLP, computer vision, and speech processing, modern methods remain siloed. Techniques engineered for one domain cannot easily transfer to another: color jitter fails on monochromatic chest X-rays, and masked token prediction does not cleanly translate to unsegmented raw sensor feeds or continuous audio spectrograms.
Developing custom SSL pipelines for the "long tail" of domains (such as biomedical datasets, multi-channel wearables, and multi-modal setups) requires extensive trial-and-error by ML specialists. DABS addresses the lack of a unified evaluation platform to test whether an SSL algorithm can learn useful representations across diverse data formats without human-designed, domain-specific adaptations.
Methods
DABS establishes strict standardization rules to measure domain agnosticism:
- Unified Backbone: A standardized 12-layer Transformer (256 hidden size, 8 heads) is kept identical across all experiments.
- Minimal Modality Adapters: Input differences are handled strictly at the boundary using minimal embedding layers: 2D patch projection for images and spectrograms, 1D patch projection for multi-channel sensor series, and standard token lookups for text.
- Strict Protocol: Domain-specific data augmentations (e.g., random resized cropping) are prohibited. Downstream evaluation is conducted via a frozen linear classifier trained on mean-pooled representation vectors.
The paper introduces two domain-agnostic SSL baseline objectives that operate directly in embedding space:
- e-Mix: A contrastive objective extending mixup to embedding space. Vectors from different examples are linearly interpolated ($\tilde{x}_i = \lambda x_i + (1-\lambda)x_{\pi(i)}$), and the model is trained with InfoNCE-style contrastive loss to predict both sources proportionally.
- ShED (Shuffled Embedding Detection): A continuous adaptation of ELECTRA. Instead of masking or replacing tokens with dictionary items, 15% of embedding positions are swapped via cyclic derangements. The network solves a binary classification task predicting which embedding vectors were shuffled.
Datasets
DABS spans seven domains, pairing an unlabeled pretraining dataset with one or more downstream transfer tasks (covering 25+ datasets in total):
- Natural Images: Pretrained on ImageNet; evaluated on CIFAR-10, DTD Textures, FGVC-Aircraft, CUB-200 Birds, GTSRB, and VGG Flowers.
- Speech Recordings: Pretrained on LibriSpeech (log-mel spectrograms); evaluated on VoxCeleb, LibriSpeech Speaker ID, Fluent Speech Commands, Google Speech Commands, and AudioMNIST.
- Monolingual English Text: Pretrained on WikiText-103; evaluated on the GLUE benchmark.
- Multilingual Text: Pretrained on an interleaved 7-language split of mC4 (EN, ES, FR, DE, ZH, JA, KO); evaluated on PAWS-X.
- Medical Imaging: Pretrained on CheXpert chest radiographs; evaluated on CheXpert's five competition pathologies and ChestX-ray8.
- Multi-channel Sensor Data: Pretrained on 52-channel PAMAP2 wearable telemetry; evaluated on physical activity recognition.
- Images with Descriptions: Pretrained on paired MS COCO image-text samples; evaluated on MS COCO retrieval and binary Visual Question Answering (VQA).
Key Insights and Ideas
- Inherited Inductive Biases Matter: Baseline success closely mirrored the algorithm’s family origin. Contrastive e-Mix achieved top performance on continuous, vision-like domains (e.g., 27.9% average accuracy on natural images vs. 20.9% for ShED), whereas prediction-based ShED excelled on structured/sequence-heavy data (e.g., 48.4 text score vs. 44.1 for e-Mix, and 88.7% sensor accuracy vs. 79.5%).
- Multimodal Failure Mode: Self-supervised pretraining on image-text pairs actually degraded performance on downstream VQA compared to an untrained, randomly initialized model (57.5% down to 48.9% with e-Mix and 54.3% with ShED), exposing the difficulty of cross-modal alignment without handcrafted multimodal objectives.
- Significant Headroom: Although generic SSL beat random baselines across unimodal tasks, accuracy gains were modest compared to specialized, domain-specific methods, confirming that domain-agnostic representation learning remains an open challenge.
Key Results
Metrics reflect average linear readout performance across each domain's transfer suite:
| Domain | Untrained Baseline | e-Mix (Contrastive) | ShED (Prediction) |
|---|---|---|---|
| Natural Images (Avg. Acc %) | 10.1% | 27.9% | 20.9% |
| Speech (Avg. Acc %) | 24.9% | 41.8% | 36.5% |
| English Text (GLUE Avg.) | 42.3 | 44.1 | 48.4 |
| Wearable Sensors (PAMAP2 Acc %) | 69.8% | 79.5% | 88.7% |
| Chest X-Rays (Avg. AUROC %) | 68.1% | 72.4% | 74.5% |
| Images & Text (VQA Acc %) | 57.5% | 48.9% | 54.3% |
Limitations and Future Work
- Domain Coverage: Although diverse, seven domains represent a fraction of scientific data types; the authors intend to add more modalities over time.
- Exclusion of Hybrid Approaches: The benchmark strictly penalizes domain-specific augmentations, meaning it does not evaluate how general-purpose objectives perform when combined with domain-specific priors.
- Architecture Constraints: Standard Transformer sequence processing with patch projection imposes minimal locality assumptions. Exploring architectures like Perceiver that entirely forgo positional patch assumptions remains an open direction.
Summary updated