Representation learning applications in biological sequence analysis

Hitoshi Iuchi, Taro Matsutani, Keisuke Yamada, Natsuki Iwano, Shunsuke Sumi, Shion Hosoda, Shitao Zhao, Tsukasa Fukunaga, Michiaki Hamada

Abstract

Remarkable advances in high-throughput sequencing have resulted in rapid data accumulation, and analyzing biological (DNA/RNA/protein) sequences to discover new insights in biology has become more critical and challenging. To tackle this issue, the application of natural language processing (NLP) to biological sequence analysis has received increased attention, because biological sequences are regarded as sentences and k-mers in these sequences as words. Embedding is an essential step in NLP, which converts words into vectors. This transformation is called representation learning and can be applied to biological sequences. Vectorized biological sequences can be used for function and structure estimation, or as inputs for other probabilistic models. Given the importance and growing trend in the application of representation learning in biology, here, we review the existing knowledge in representation learning for biological sequence analysis.

Introduction

The explosion of high-throughput sequencing has created a massive data bottleneck: we can read genetic code faster than we can understand it. Traditional analysis often relies on sequence alignment—comparing how similar two strings of letters are—but this approach frequently fails to capture the underlying "grammar" of biology. This grammar consists of complex biophysical and biochemical rules that determine how a sequence folds or functions.

The challenge lies in creating "distributed representations" (vectors) of these sequences. Just as NLP allows a computer to understand that a word's meaning changes based on context, biological representation learning aims to help models understand how a specific amino acid's role changes based on its neighbors.

Methodological Evolution

The field has evolved from simple static embeddings to complex, context-aware architectures that treat biological sequences as a language to be decoded.

From Word2vec to Contextualized RNNs

The foundational approach involves breaking sequences into k-mers (short, overlapping substrings), which are treated as "words." Models like ProtVec use the distributional hypothesis to map these k-mers into vector space. To address the "polysemy" problem—where the same sequence has different functions in different environments—models like SeqVec and UniRep utilize Recurrent Neural Networks (RNNs) and LSTMs. These process sequences step-by-step, allowing the "meaning" of a residue to be influenced by its preceding sequence.

Transformers and Attention Mechanisms

The current state-of-the-art involves Transformers, which use attention mechanisms to weigh the importance of different parts of a sequence simultaneously. This is particularly powerful for biology because it can capture "distal interactions"—residues that are far apart in a linear sequence but touch each other when a protein folds into 3D space.

Architecture of Transformer-based models like BERT applied to biological sequences using masking and next-sentence prediction tasks.

Key Insights and Applications

Representation learning allows for "pre-training" on massive, unlabeled datasets (like UniRef100 with 250 million sequences). A model can learn the general rules of proteins and then be "fine-tuned" on small, specialized datasets to achieve high accuracy in specific tasks.

One of the most striking findings is that "attention maps" in models like ESM-1b often correlate with actual physical contacts in a protein's 3D structure, despite the models being trained only on 1D text strings. Furthermore, methods like CSCS can identify "escape mutations" in viruses by modeling evolutionary semantic changes.

Survey of Results

As summarized in the table below, representation learning has been successfully applied across a wide range of biological tasks, from structure prediction to synthetic biology design.

Comprehensive survey of representation learning methods, models, training data, and biological tasks.
  • Structure Prediction: Models like TAPE and ProtBert achieve state-of-the-art results in predicting secondary structures and contact maps.
  • Functional Annotation: UDSMProt has outperformed traditional methods in enzyme class prediction.
  • Genomic Regulation: DNABERT demonstrated high performance in identifying transcription start sites, showing that models trained on human DNA can generalize to other organisms.

Limitations and Future Directions

Despite significant progress, several hurdles remain. Biological sequences often exceed the typical 512-token limit of standard BERT models, and the 20-amino-acid vocabulary of proteins is relatively small compared to natural languages. Future work may explore hyperbolic embeddings to better represent hierarchical evolutionary relationships and emphasize the release of pre-trained models to ensure reproducibility across the research community.

Summary updated

Abstract

Although remarkable advances have been reported in high-throughput sequencing, the ability to aptly analyze a substantial amount of rapidly generated biological (DNA/RNA/protein) sequencing data remains a critical hurdle. To tackle this issue, the application of natural language processing (NLP) to biological sequence analysis has received increased attention. In this method, biological sequences are regarded as sentences while the single nucleic acids/amino acids or k-mers in these sequences represent the words. Embedding is an essential step in NLP, which performs the conversion of these words into vectors. Specifically, representation learning is an approach used for this transformation process, which can be applied to biological sequences. Vectorized biological sequences can then be applied for function and structure estimation, or as input for other probabilistic models. Considering the importance and growing trend for the application of representation learning to biological research, in the present study, we have reviewed the existing knowledge in representation learning for biological sequence analysis.
BibTeX
@inproceedings{iuchi2021representation,
  author = {Hitoshi Iuchi and Taro Matsutani and Keisuke Yamada and Natsuki Iwano and Shunsuke Sumi and Shion Hosoda and Shitao Zhao and Tsukasa Fukunaga and Michiaki Hamada},
  title = {{Representation learning applications in biological sequence analysis}},
  booktitle = {Computational},
  year = {2021},
  doi = {10.1016/j.csbj.2021.05.039},
}
Figure