Why can a machine learning model trained on English text predict gene expression, parse RNA structures, or design therapeutic antibodies? At first glance, human language and molecular biology seem entirely distinct. However, a growing body of research demonstrates that biological sequences (DNA, RNA, and proteins) share deep statistical, structural, and hierarchical properties with natural language.
By treating the genome or proteome as a formal language, researchers can "reprogram" pretrained natural language models to solve complex biological tasks. This cross-modal transfer often outperforms specialized models trained from scratch on biological data alone, leveraging the universal structural reasoning capabilities embedded within large language models (LLMs).
The Shared Grammar of Language and Biology
The fundamental reason language models transfer effectively to biology is that both domains are governed by a strict, hierarchical syntax. A sequence of nucleotides or amino acids is not a random string; it contains functional units (like genes or structural domains) composed of specific sub-elements, much like sentences are built from phrases and words.
This linguistic analogy was first formalized by Richard Durbin in Interpreting the Human Genome Sequence, Using Stochastic Grammars, where computational linguistics frameworks like stochastic grammars were adapted to automate gene identification and protein family classification. The structural parallels manifest in several key ways:
- Zipfian Distributions: Both human languages and protein sequences exhibit a Zipfian distribution, where a small number of tokens appear with extremely high frequency while the vast majority are rare. As noted by Pin-Yu Chen et al. in Reprogramming pretrained language models for protein sequence representation learning, this shared statistical regularity allows models trained on text to generalize their understanding of sequence distribution and token relationships to biological data.
- Repetitive and Informational Structures: The distribution of repetitive elements in genomic sequences mirrors the structure of human language, where functional or structural components show high repetition while information-dense components are more unique (Nava Whiteford et al. in Visualizing the Repeat Structure of Genomic Sequences).
- Structural Dependencies: In language, words depend on each other across long distances (e.g., subject-verb agreement). In biology, physical interactions create similar long-range dependencies, such as base-pairing in RNA folding. The authors of DEPfold: RNA Secondary Structure Prediction as Dependency Parsing. exploited this by framing RNA secondary structure prediction as a natural language dependency parsing task, demonstrating that the underlying logic of sequence dependencies transfers directly from text to molecular biology.
This bidirectional relationship is so pronounced that the structural alignment algorithms can even flow the other way: Jianyong Duan et al. in A Bio-Inspired Approach for Multi-Word Expression Extraction adapted gene sequence alignment algorithms from bioinformatics to identify flexible multi-word expressions in natural language, treating sentences like DNA strands prone to insertions and mutations.
Mechanics of Cross-Modal Model Reprogramming
To adapt a frozen language model to biological sequences without expensive retraining, researchers use model reprogramming or token alignment. The goal is to map biological entities (such as amino acids or DNA k-mers) into the language model's existing token embedding space.
[Biological Sequence] -> [Linear Projection / Dictionary Learning] -> [Linguistic Token Space] -> [Frozen LLM]
In Reprogramming Large Pretrained Language Models for Antibody Sequence Infilling, Igor Melnyk et al. introduced ReprogBert, a framework that repurposes a frozen English BERT model for antibody design. By training only a small set of linear projection matrices to map amino acids to English tokens, the model generates highly diverse and structurally sound antibody sequences.
Similarly, Edouardo Honig et al. developed Genetic sequence Token Alignment (GTA) in Long-range gene expression prediction with token alignment of large language model. GTA aligns genomic features with the embedding space of a large language model (like Llama3), allowing the model to use its pretrained reasoning capabilities to predict gene expression across context windows as large as 1 million base pairs.
To overcome the small alphabet size of DNA (only 4 bases), researchers often apply classic NLP tokenization strategies. For instance, the authors of An Embarrassingly Simple Approach to Enhance Transformer Performance in Genomic Selection for Crop Breeding utilized k-mer tokenization and random masking on genomic sequences, allowing standard Transformer architectures to capture complex genetic markers efficiently.
Why Pretrained Text Models Serve as Universal Priors
A compelling question remains: why use a model trained on text rather than training a model exclusively on biological data from scratch?
The answer lies in the nature of the representations learned during large-scale pretraining. Natural language corpora are massive, highly diverse, and rich in complex hierarchical structures. Training on text forces a model to develop sophisticated inductive biases for handling long-range dependencies, hierarchical nesting, and contextual meaning.
Wei-Tsung Kao et al. investigated this phenomenon in Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability, finding that models pretrained on natural language possess a latent capacity for cross-disciplinary transfer that allows them to outperform models trained from scratch on non-linguistic sequences. Pretraining captures abstract structural and logical patterns that are not unique to human communication.
This capacity for abstract structural learning is further supported by Isabel Papadimitriou et al. in Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models, which demonstrated that pretraining on non-linguistic structured data (like MIDI music or Java code) significantly enhances a model's subsequent acquisition of natural language grammar. As synthesized by Mikhail Budnikov et al. in Generalization potential of large language models, pretraining on highly structured symbolic domains teaches models universal structural principles that generalize across disparate symbolic systems.
When applied to biology, frameworks like ORCA (Junhong Shen et al. in Cross-Modal Fine-Tuning: Align then Refine) and L2G (Wenduo Cheng et al. in L2G: Repurposing Language Models for Genomics Tasks) exploit these universal priors. The authors of L2G demonstrated that adapting a text-pretrained model like RoBERTa to genomic sequence classification can outperform massive, purpose-built genomic foundation models while requiring orders of magnitude less data and compute. The structural reasoning capabilities—such as tracking long-range regulatory interactions in DNA—are already present in the language model; they merely need to be activated through proper alignment.
Key Foundational Work
Invited Talk: Interpreting the Human Genome Sequence, Using Stochastic Grammars (2001) — contributed 2 concept elements to this direction.
Interpreting the Human Genome Sequence, Using Stochastic Grammars (2001) — contributed 2 concept elements to this direction.
A Bio-Inspired Approach for Multi-Word Expression Extraction (2006) — contributed 1 concept elements to this direction.