Many early genomic language models focused exclusively on a single reference genome, typically the human genome. However, as noted by Yair Schiff et al. in Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling, this narrow focus severely limits a model's capacity to capture broader evolutionary patterns.
Training on a single genome introduces a single-species bias. Without seeing alternative variations, a model cannot easily distinguish between highly critical regulatory elements and neutral, non-functional "noise" that just happens to be present in that specific genome. As Yu Bo et al. argue in Revisiting Convolution Architecture in the Realm of DNA Foundation Models, single-species training restricts a model's ability to generalize across diverse biological contexts. Alice Del Vecchio et al. echo this in PatchDNA: A Flexible and Biologically-Informed Alternative to Tokenization for DNA, stating that multi-species data is essential to fully leverage evolutionary conservation signals.
Evolution as an Unsupervised Grammar
Why does mixing genomes from humans, mice, plants, and microbes work? The answer lies in how evolution acts as a natural statistical filter. Over millions of years, mutations in vital genomic regions (such as promoters or protein-coding sequences) are typically deleterious, meaning organisms carrying them are less likely to survive. Consequently, these functional sequences are "conserved" across species. Conversely, non-functional regions mutate freely.
By training a language model on a diverse, multi-species corpus, the model implicitly learns this "evolutionary grammar." In Genomic Language Models: Opportunities and Challenges, Gonzalo Benegas et al. explain that cross-species training leverages phylogenetic information to help the model decode the underlying syntax of functional DNA. Instead of relying on explicit, computationally expensive multiple sequence alignments, the model learns to recognize conserved motifs simply by trying to predict masked nucleotides across millions of diverse sequences.
This exposure provides what Zhihan Zhou et al. describe in DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes as a "richer structural vocabulary." By observing how nature varies or preserves specific sequences across taxa, the model develops a robust understanding of universal genomic principles.
The Cross-Species Paradox
One might assume that to predict human gene expression or variant effects, a model should train exclusively on human data to avoid being confused by other organisms. Empirically, the opposite is true.
In The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics, Hugo Dalla-Torre et al. demonstrated that pre-training on diverse multi-species genomic data actually enhances performance on human-specific tasks compared to training on human data alone. This occurs because the fundamental regulatory logic is shared across the tree of life. As Žiga Avsec et al. showed with Enformer in Effective gene expression prediction from sequence by integrating long-range interactions, multitask learning across multiple species allows models to leverage this shared regulatory logic to build superior, generalized biological representations.
A comprehensive benchmark by Haonan Feng et al. in Benchmarking DNA foundation models for genomic and genetic tasks confirmed this effect, showing that cross-species evolutionary diversity is a primary driver of performance and generalization for DNA foundation models. By learning from a wide variety of organisms, the model avoids overfitting to single-genome idiosyncrasies.
Generalizing to Unseen Species
The ultimate test of a foundation model's understanding is its ability to generalize to organisms it has barely seen, or not seen at all. Multi-species pre-training enables true zero-shot generalization by mapping sequences to a universal biological grammar.
For example, the authors of A foundational large language model for edible plant genomes trained a transformer model (AgroNT) on dozens of plant species and found that capturing these evolutionary conserved patterns allowed the model to generalize to under-researched "orphan crops" where experimental data is virtually non-existent. Similarly, Kepeng Lin et al. demonstrated in PlantBiMoE: A Bidirectional Foundation Model with SparseMoE for Plant Genomes that large-scale multi-species pre-training captures fundamental evolutionary patterns that transcend individual species boundaries, allowing the model to adapt to specific species not emphasized during training.
This strategy is also highly effective when dealing with rapidly mutating systems or specialized domains. In GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics, Maxim Zvyagin et al. utilized a tiered training strategy, first pre-training on a massive, diverse corpus of prokaryotic gene sequences to learn a general biological grammar before fine-tuning on specific viral datasets. This broad evolutionary foundation allowed the model to better capture the underlying constraints governing viral dynamics.
Key Foundational Work
Effective gene expression prediction from sequence by integrating long-range interactions (2021) — contributed 1 concept elements to this direction.
DNA language models are powerful predictors of genome-wide variant effects (2022) — contributed 1 concept elements to this direction.
Language models enable zero-shot prediction of the effects of mutations on protein function (2021) — contributed 1 concept elements to this direction.