DNABERT-S: Pioneering Species Differentiation with Species-Aware DNA Embeddings

Zhihan Zhou, Weimin Wu, Harrison Ho, Jiayi Wang, Lizhen Shi, Ramana V Davuluri, Zhong Wang, Han LiuarXiv 2024 · arXiv

TL;DR

DNABERT-S adapts a genome foundation model to generate "species-aware" DNA embeddings that naturally cluster and segregate sequences from different organisms without relying on reference genomes. By combining a two-phase curriculum contrastive learning strategy with latent-space representation mixing, it doubles the performance of traditional and deep-learning baselines in unsupervised species clustering and metagenomic binning.

What Problem Does This Paper Address?

In fields like microbial ecology and epidemiology, researchers often sequence DNA directly from complex environmental samples (e.g., soil, seawater, or the human gut) containing thousands of uncharacterized species. Grouping these fragmented DNA sequences by their species of origin—a critical process known as metagenomics binning—is incredibly challenging because the vast majority of these organisms lack well-documented reference genomes.

While recent genome foundation models (large language models trained on DNA) have excelled at functional genomic tasks, their standard pre-training objectives (like masked language modeling) do not incentivize the model to separate sequences by species. Surprisingly, vanilla foundation models often underperform simple, non-trainable statistical methods like Tetra-Nucleotide Frequency (TNF). DNABERT-S bridges this gap by explicitly training a foundation model to learn the subtle, global signatures that differentiate species.

Methods

DNABERT-S builds upon the pre-trained DNABERT-2 architecture and introduces a specialized training framework designed to map DNA fragments into a species-segregated embedding space:

  1. Curriculum Contrastive Learning ($C^2\text{LR}$): The model is trained in two distinct phases that progressively increase in difficulty. Phase I uses a Weighted SimCLR objective, where non-overlapping DNA fragments from the same genome act as positive pairs, while fragments from different genomes serve as negative pairs. Harder negatives (those closer in the embedding space) are given higher weights.
  2. Manifold Instance Mixup (MI-Mix): Introduced in Phase B, this technique addresses a unique challenge in genomic data augmentation. While mixing raw text strings works for natural language, mixing raw DNA strings lacks biological plausibility. Instead, MI-Mix interpolates the intermediate hidden states (embeddings) of different sequences at a randomly selected layer of the model. The network is then tasked with differentiating these mixed proportions at the output layer, forcing it to learn smoother and more robust decision boundaries between species.
Overview of DNABERT-S training process, highlighting the data construction, two-phase curriculum learning, and the Manifold Instance Mixup mechanism.

Datasets

  • Training Data: Constructed using reference genomes from GenBank, consisting of 2 million randomly selected non-overlapping DNA sequence pairs (10,000 base pairs each) spanning 17,636 viral, 5,011 fungal, and 6,402 bacterial genomes.
  • Evaluation Benchmarks: Evaluated across 23 diverse datasets:
  • 14 highly imbalanced long-read datasets from the CAMI2 (Critical Assessment of Metagenome Interpretation) challenge, representing Marine and Plant-associated environments.
  • 9 synthetic datasets derived from reference genomes.
  • 3 synthetic datasets containing completely unseen non-microbe categories (mammals, invertebrates, and protozoa) to evaluate the model's out-of-domain generalizability.

Key Insights and Ideas

  • Foundation Models Aren't a Free Lunch for Clustering: Standard language modeling objectives do not inherently structure embedding spaces for clustering or retrieval. Without explicit contrastive fine-tuning, massive neural networks can lose to simple k-mer counting methods.
  • The Latent Space is Better for Augmentation: For discrete biological data like DNA, performing data mixup in the latent manifold space provides a powerful, smooth regularization signal that text-level mutations cannot replicate.
  • The Specialization Trade-off: Forcing a model to prioritize species-specific features inherently reduces its performance on general genomic function prediction tasks (such as predicting promoters or epigenetic marks). The authors note this is a deliberate and acceptable trade-off for a specialized tool.

Key Results

  • Clustering and Binning Superiority: DNABERT-S doubled the Adjusted Rand Index (ARI) in K-Means species clustering compared to the strongest existing baseline. In metagenomics binning simulations, it successfully recovered twice as many unique species with high F1 scores (>0.5) than traditional methods.
Figure 3: Metagenomics Binning Results. The bin size represents the number of unique species identified by each model and different colors represent the F1 score of the identified species. With high F1 scores, DNABERT-S identifies many more species than the baselines.
  • Few-Shot Efficiency: In species classification tasks, a linear classifier trained on DNABERT-S embeddings with just 2 labeled samples per class (2-shot) outperformed the top baseline trained with 10 samples per class (10-shot).
  • Extreme Dimension Resilience: The embeddings proved remarkably robust to compression. When the 768-dimensional embeddings were reduced to just 16 dimensions using simple average pooling, DNABERT-S still outperformed the standard 256-dimensional TNF baseline.

Limitations and Future Work

  • High Computational Demand: Deep learning models are computationally heavy. While traditional methods like TNF run in seconds on a CPU, training DNABERT-S required roughly 48 hours on 8 NVIDIA A100 GPUs, and inference remains significantly slower than traditional alignment tools like MMseqs2.
  • Domain Expansion: While the model showed promising transferability to non-microbe species (like mammals), the absolute performance was lower due to higher genetic similarities among mammals. Future work includes developing in-domain, species-aware models for specific non-microbial taxonomic groups.

Summary updated

Abstract

We introduce DNABERT-S, a tailored genome model that develops species-aware embeddings to naturally cluster and segregate DNA sequences of different species in the embedding space. Differentiating species from genomic sequences (i.e., DNA and RNA) is vital yet challenging, since many real-world species remain uncharacterized, lacking known genomes for reference. Embedding-based methods are therefore used to differentiate species in an unsupervised manner. DNABERT-S builds upon a pre-trained genome foundation model named DNABERT-2. To encourage effective embeddings to error-prone long-read DNA sequences, we introduce Manifold Instance Mixup (MI-Mix), a contrastive objective that mixes the hidden representations of DNA sequences at randomly selected layers and trains the model to recognize and differentiate these mixed proportions at the output layer. We further enhance it with the proposed Curriculum Contrastive Learning (C$^2$LR) strategy. Empirical results on 23 diverse datasets show DNABERT-S's effectiveness, especially in realistic label-scarce scenarios. For example, it identifies twice more species from a mixture of unlabeled genomic sequences, doubles the Adjusted Rand Index (ARI) in species clustering, and outperforms the top baseline's performance in 10-shot species classification with just a 2-shot training. Model, codes, and data are publicly available at \url{https://github.com/MAGICS-LAB/DNABERT_S}.
BibTeX
@misc{zhou2024dnabert,
  author = {Zhihan Zhou and Weimin Wu and Harrison Ho and Jiayi Wang and Lizhen Shi and Ramana V Davuluri and Zhong Wang and Han Liu},
  title = {{DNABERT-S: Pioneering Species Differentiation with Species-Aware DNA Embeddings}},
  year = {2024},
  eprint = {2402.08777},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.08777},
}

Loading related content...

Figure