Self-supervised learning (SSL) has revolutionized how we train AI on text and images, but does a "one-size-fits-all" algorithm actually exist for the rest of the world’s data? In DABS 2.0, researchers from Stanford University expand their "Domain-Agnostic Benchmark for Self-Supervision" to find out, pushing SSL into the realms of genomics, particle physics, and satellite imagery.
Overview
DABS 2.0 expands a universal benchmark for self-supervised learning to 12 diverse domains, including five new scientific and engineering fields. The paper introduces new algorithms and reveals that the "optimal" way to train these models—specifically how much data you should hide or corrupt during training—depends heavily on the inherent redundancy of the domain.
The Challenge of Domain-Specific AI
Most breakthroughs in SSL (like BERT or MAE) are developed in silos: one group works on text, another on natural images, and another on speech. This leads to domain-specific "recipes" that might not work for a biologist studying protein sequences or an engineer monitoring semiconductor manufacturing.
The authors argue that for AI to be truly useful in specialized scientific fields where labeled data is scarce, we need universal SSL algorithms. The challenge is that different data types have different "textures." An image has high redundancy (neighboring pixels are often similar), while a DNA sequence is dense with information. DABS 2.0 provides a standardized playground to see if we can develop a single architecture and objective that generalizes across these vastly different data landscapes.
Methods
The researchers utilize a Domain Agnostic Transformer architecture, which uses specialized "embedding modules" to turn any input (pixels, tokens, or sensor readings) into a sequence of vectors that a standard Transformer can process. They evaluate three primary SSL strategies:
- Generalized Masked Autoencoding (MAE): A generalization of the popular MAE method. For discrete data (like text or DNA), it uses a softmax/cross-entropy loss; for continuous data (like sensor readings), it uses Mean Squared Error (MSE) to reconstruct masked parts of the input.
- Capri (Contrastive Prediction of Redacted Embeddings): A new hybrid algorithm. Instead of trying to reconstruct the exact values of a masked section, Capri tries to predict an embedding that is "closer" (via cosine similarity) to the original than to other parts of the sequence.
- ShED (Shuffled Embedding Detection): A baseline that permutes a fraction of the input embeddings and trains the model to identify which ones were swapped.
Datasets
The benchmark now spans 12 domains.
The five new additions in version 2.0 are:
- Bacterial Genomics: 1M DNA sequences (A, C, G, T tokens).
- Semiconductor Wafers: Grayscale images of defect patterns in manufacturing.
- Particle Physics: Tabular data from the HIGGS dataset (Monte Carlo simulations).
- Protein Biology: 31M sequences from the Pfam database.
- Multispectral Satellite Imagery: 13-channel images from the EuroSAT dataset.
These join the original seven: Wearable Sensors, English/Multilingual Text, Speech, Chest X-rays, Paired Image+Text, and Natural Images.
Key Insights: The Redundancy Hypothesis
The most significant insight from the paper is the Redundancy Hypothesis. The authors found that the optimal "corruption rate" (how much data you mask or shuffle) varies wildly depending on the dataset. In "redundant" domains like satellite imagery, you need to mask a lot (up to 85%) to force the model to learn useful features. In "dense" domains like genomics or text, masking too much destroys the signal, and lower rates (15%) work better.
Furthermore, the researchers noted a "Softmax Bottleneck." For tokenized datasets with huge vocabularies, MAE often ran into Out-of-Memory (OM) errors, highlighting a practical limitation of reconstruction-based methods for specialized scientific languages.
Key Results
The authors measured success by "transfer accuracy"—how well a model pretrained with SSL performs on a labeled task compared to a model trained from scratch ("None").
While MAE and ShED both showed significant gains over training from scratch, no single algorithm dominated every domain:
- Satellite Imagery: MAE (85% masking) achieved 84.8% accuracy, a massive jump from the 62.3% baseline.
- Genomics: MAE (50% masking) reached 39.0% accuracy, nearly doubling the 22.4% baseline.
- Particle Physics: ShED (15% corruption) hit 68.0%, significantly outperforming the 54.8% baseline.
- Capri's Struggles: Despite being a novel hybrid, Capri often diverged during training or performed worse than the simpler ShED and MAE methods.
Limitations and Future Work
The study primarily focused on linear evaluation (freezing the model and training a simple classifier on top). While this is a standard way to measure feature quality, many SSL models perform best when "fine-tuned" (updating the whole model).
The authors also acknowledge that 12 domains, while impressive, represent only a small slice of the real world. Future work involves scaling these algorithms to even larger models and more complex data types like 3D point clouds or graphs, and finding an automated way to determine the optimal corruption rate for a new domain without expensive trial-and-error.
Summary updated