DABS 2.0: Improved Datasets and Algorithms for Universal Self-Supervision

Alex Tamkin, Gaurab Banerjee, Mohamed Owda, Vincent Liu, Shashank Rammoorthy, Noah D. GoodmanNeurIPS 2022 · OpenReview

Self-supervised learning (SSL) has revolutionized how we train AI on text and images, but does a "one-size-fits-all" algorithm actually exist for the rest of the world’s data? In DABS 2.0, researchers from Stanford University expand their "Domain-Agnostic Benchmark for Self-Supervision" to find out, pushing SSL into the realms of genomics, particle physics, and satellite imagery.

Overview

DABS 2.0 expands a universal benchmark for self-supervised learning to 12 diverse domains, including five new scientific and engineering fields. The paper introduces new algorithms and reveals that the "optimal" way to train these models—specifically how much data you should hide or corrupt during training—depends heavily on the inherent redundancy of the domain.

The Challenge of Domain-Specific AI

Most breakthroughs in SSL (like BERT or MAE) are developed in silos: one group works on text, another on natural images, and another on speech. This leads to domain-specific "recipes" that might not work for a biologist studying protein sequences or an engineer monitoring semiconductor manufacturing.

The authors argue that for AI to be truly useful in specialized scientific fields where labeled data is scarce, we need universal SSL algorithms. The challenge is that different data types have different "textures." An image has high redundancy (neighboring pixels are often similar), while a DNA sequence is dense with information. DABS 2.0 provides a standardized playground to see if we can develop a single architecture and objective that generalizes across these vastly different data landscapes.

Methods

The researchers utilize a Domain Agnostic Transformer architecture, which uses specialized "embedding modules" to turn any input (pixels, tokens, or sensor readings) into a sequence of vectors that a standard Transformer can process. They evaluate three primary SSL strategies:

  1. Generalized Masked Autoencoding (MAE): A generalization of the popular MAE method. For discrete data (like text or DNA), it uses a softmax/cross-entropy loss; for continuous data (like sensor readings), it uses Mean Squared Error (MSE) to reconstruct masked parts of the input.
  2. Capri (Contrastive Prediction of Redacted Embeddings): A new hybrid algorithm. Instead of trying to reconstruct the exact values of a masked section, Capri tries to predict an embedding that is "closer" (via cosine similarity) to the original than to other parts of the sequence.
  3. ShED (Shuffled Embedding Detection): A baseline that permutes a fraction of the input embeddings and trains the model to identify which ones were swapped.

Datasets

The benchmark now spans 12 domains.

DABS 2.0 expands the benchmark to 12 diverse domains, adding five new scientific and engineering fields to the original DABS 1.0 dataset.

The five new additions in version 2.0 are:

  • Bacterial Genomics: 1M DNA sequences (A, C, G, T tokens).
  • Semiconductor Wafers: Grayscale images of defect patterns in manufacturing.
  • Particle Physics: Tabular data from the HIGGS dataset (Monte Carlo simulations).
  • Protein Biology: 31M sequences from the Pfam database.
  • Multispectral Satellite Imagery: 13-channel images from the EuroSAT dataset.

These join the original seven: Wearable Sensors, English/Multilingual Text, Speech, Chest X-rays, Paired Image+Text, and Natural Images.

Key Insights: The Redundancy Hypothesis

The most significant insight from the paper is the Redundancy Hypothesis. The authors found that the optimal "corruption rate" (how much data you mask or shuffle) varies wildly depending on the dataset. In "redundant" domains like satellite imagery, you need to mask a lot (up to 85%) to force the model to learn useful features. In "dense" domains like genomics or text, masking too much destroys the signal, and lower rates (15%) work better.

Furthermore, the researchers noted a "Softmax Bottleneck." For tokenized datasets with huge vocabularies, MAE often ran into Out-of-Memory (OM) errors, highlighting a practical limitation of reconstruction-based methods for specialized scientific languages.

Key Results

The authors measured success by "transfer accuracy"—how well a model pretrained with SSL performs on a labeled task compared to a model trained from scratch ("None").

Transfer accuracy across domains for ShED, Capri, and MAE at varying corruption rates. Bold numbers indicate the best performance per domain.

While MAE and ShED both showed significant gains over training from scratch, no single algorithm dominated every domain:

  • Satellite Imagery: MAE (85% masking) achieved 84.8% accuracy, a massive jump from the 62.3% baseline.
  • Genomics: MAE (50% masking) reached 39.0% accuracy, nearly doubling the 22.4% baseline.
  • Particle Physics: ShED (15% corruption) hit 68.0%, significantly outperforming the 54.8% baseline.
  • Capri's Struggles: Despite being a novel hybrid, Capri often diverged during training or performed worse than the simpler ShED and MAE methods.

Limitations and Future Work

The study primarily focused on linear evaluation (freezing the model and training a simple classifier on top). While this is a standard way to measure feature quality, many SSL models perform best when "fine-tuned" (updating the whole model).

The authors also acknowledge that 12 domains, while impressive, represent only a small slice of the real world. Future work involves scaling these algorithms to even larger models and more complex data types like 3D point clouds or graphs, and finding an automated way to determine the optimal corruption rate for a new domain without expensive trial-and-error.

Summary updated

Abstract

Universal self-supervised (SSL) algorithms hold enormous promise for making machine learning accessible to high-impact domains such as protein biology, manufacturing, and genomics. We present DABS 2.0: a set of improved datasets and algorithms for advancing research on universal SSL. We extend the recently-introduced DABS benchmark with the addition of five real-world science and engineering domains: protein biology, bacterial genomics, multispectral satellite imagery, semiconductor wafers, and particle physics, bringing the total number of domains in the benchmark to twelve. We also propose a new universal SSL algorithm, Capri, and a generalized version of masked autoencoding, and apply both on all twelve domains---the most wide-ranging exploration of SSL yet. We find that multiple algorithms show gains across domains, outperforming previous baselines. In addition, we demonstrate the usefulness of DABS for scientific study of SSL by investigating the optimal corruption rate for each algorithm, showing that the best setting varies based on the domain. Code will be released at http://github.com/alextamkin/dabs}{http://github.com/alextamkin/dabs
BibTeX
@inproceedings{tamkin2022dabs,
  author = {Alex Tamkin and Gaurab Banerjee and Mohamed Owda and Vincent Liu and Shashank Rammoorthy and Noah D. Goodman},
  title = {{DABS 2.0: Improved Datasets and Algorithms for Universal Self-Supervision}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=jmd-hKt4zDA},
}

Loading related content...

Figure