Abstract
Remarkable advances in high-throughput sequencing have resulted in rapid data accumulation, and analyzing biological (DNA/RNA/protein) sequences to discover new insights in biology has become more critical and challenging. To tackle this issue, the application of natural language processing (NLP) to biological sequence analysis has received increased attention, because biological sequences are regarded as sentences and k-mers in these sequences as words. Embedding is an essential step in NLP, which converts words into vectors. This transformation is called representation learning and can be applied to biological sequences. Vectorized biological sequences can be used for function and structure estimation, or as inputs for other probabilistic models. Given the importance and growing trend in the application of representation learning in biology, here, we review the existing knowledge in representation learning for biological sequence analysis.
Introduction
The explosion of high-throughput sequencing has created a massive data bottleneck: we can read genetic code faster than we can understand it. Traditional analysis often relies on sequence alignment—comparing how similar two strings of letters are—but this approach frequently fails to capture the underlying "grammar" of biology. This grammar consists of complex biophysical and biochemical rules that determine how a sequence folds or functions.
The challenge lies in creating "distributed representations" (vectors) of these sequences. Just as NLP allows a computer to understand that a word's meaning changes based on context, biological representation learning aims to help models understand how a specific amino acid's role changes based on its neighbors.
Methodological Evolution
The field has evolved from simple static embeddings to complex, context-aware architectures that treat biological sequences as a language to be decoded.
From Word2vec to Contextualized RNNs
The foundational approach involves breaking sequences into k-mers (short, overlapping substrings), which are treated as "words." Models like ProtVec use the distributional hypothesis to map these k-mers into vector space. To address the "polysemy" problem—where the same sequence has different functions in different environments—models like SeqVec and UniRep utilize Recurrent Neural Networks (RNNs) and LSTMs. These process sequences step-by-step, allowing the "meaning" of a residue to be influenced by its preceding sequence.
Transformers and Attention Mechanisms
The current state-of-the-art involves Transformers, which use attention mechanisms to weigh the importance of different parts of a sequence simultaneously. This is particularly powerful for biology because it can capture "distal interactions"—residues that are far apart in a linear sequence but touch each other when a protein folds into 3D space.
Key Insights and Applications
Representation learning allows for "pre-training" on massive, unlabeled datasets (like UniRef100 with 250 million sequences). A model can learn the general rules of proteins and then be "fine-tuned" on small, specialized datasets to achieve high accuracy in specific tasks.
One of the most striking findings is that "attention maps" in models like ESM-1b often correlate with actual physical contacts in a protein's 3D structure, despite the models being trained only on 1D text strings. Furthermore, methods like CSCS can identify "escape mutations" in viruses by modeling evolutionary semantic changes.
Survey of Results
As summarized in the table below, representation learning has been successfully applied across a wide range of biological tasks, from structure prediction to synthetic biology design.
- Structure Prediction: Models like TAPE and ProtBert achieve state-of-the-art results in predicting secondary structures and contact maps.
- Functional Annotation: UDSMProt has outperformed traditional methods in enzyme class prediction.
- Genomic Regulation: DNABERT demonstrated high performance in identifying transcription start sites, showing that models trained on human DNA can generalize to other organisms.
Limitations and Future Directions
Despite significant progress, several hurdles remain. Biological sequences often exceed the typical 512-token limit of standard BERT models, and the 20-amino-acid vocabulary of proteins is relatively small compared to natural languages. Future work may explore hyperbolic embeddings to better represent hierarchical evolutionary relationships and emphasize the release of pre-trained models to ensure reproducibility across the research community.
Summary updated