Tokenizing Biological Data for Transformer Architectures

52 concepts47 papers1 proposal

To apply modern Transformer architectures to biology, researchers must solve a fundamental mismatch in data representation. Transformers are designed to process sequences of discrete tokens chosen from a predefined vocabulary, like words or subwords in human language. Biological data, however, arrives either as long, continuous strings of individual characters (such as the nucleotides A, C, G, T) or as high-dimensional, continuous matrices of cellular gene expression.

Tokenization is the critical translation layer that converts these raw biological signals into a structured vocabulary that models can interpret. As reviewed by Shuyan Cheng et al. in Deciphering genomic codes using advanced natural language processing techniques: a scoping review, selecting the right tokenization strategy determines whether a model can efficiently capture long-range biological syntax or get bogged down by computational overhead and technical noise.

Fixed-Length k-mers and the Challenge of Shift Sensitivity

The earliest and most straightforward approach to tokenizing genetic sequences is the $k$-mer strategy. Instead of feeding individual nucleotides into a model one by one—which provides too little context per step—the sequence is sliced into strings of length $k$. For example, a DNA sequence like ATCGAT can be broken into overlapping 3-mers: ATC, TCG, CGA, and GAT.

This approach works because it creates a richer vocabulary that inherently captures local structural motifs and regulatory patterns. Monica F. Danilevicz et al. leveraged overlapping $k$-mers in DNABERT-based explainable lncRNA identification in plant genome assemblies to map local sequence dependencies without relying on handcrafted features. Similarly, Monireh Safari et al. utilized fixed 4-mer and 6-mer tokens in Enhancing DNA Foundation Models to Address Masking Inefficiencies to build discrete representations for DNA barcodes.

To process longer sequences within standard attention memory constraints, some architectures opt for non-overlapping $k$-mers. As demonstrated by Renqi Chen et al. in An Embarrassingly Simple Approach to Enhance Transformer Performance in Genomic Selection for Crop Breeding, grouping consecutive genetic markers into non-overlapping chunks significantly compresses the total sequence length. This compression reduces the quadratic computational complexity of self-attention, making long-range dependencies trackable.

However, fixed-length $k$-mers suffer from a major flaw: shift sensitivity. If a single nucleotide is deleted or inserted due to a mutation or sequencing error, every subsequent fixed-length window changes completely. An insertion shifts the reading frame, altering every downstream token and preventing the model from recognizing conserved biological patterns further down the strand.

Dynamic Subword Tokenization via Byte Pair Encoding

To overcome shift sensitivity and compress sequences more intelligently, researchers adapted Byte Pair Encoding (BPE)—the standard subword tokenization algorithm used in large language models like GPT—for biological sequences.

Instead of slicing DNA into arbitrary chunks of fixed length, BPE builds a vocabulary iteratively based on statistics. It starts with individual characters and progressively merges the most frequent pairs. In a genomic context, highly conserved, recurring motifs (such as common promoter regions or transcription factor binding sites) are merged into single, variable-length tokens, while rare or mutated sequences remain broken down into smaller pieces.

Fixed 3-mer (Overlapping):  [ATG] [TGC] [GCA] [CAT]
Byte Pair Encoding (BPE):   [ATGC] [A] [TG] [CCGATA]
                            (Conserved motifs become single tokens)

This shift from fixed to dynamic tokenization fundamentally changes how models process genetic syntax. Zhihan Zhou et al. demonstrated in DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes that BPE drastically reduces sequence lengths and computational overhead. Crucially, the authors showed that BPE mitigates shift sensitivity: a single nucleotide deletion only alters tokens locally where the edit occurred, leaving the downstream subword representations intact.

This statistical compression is vital for scaling models to complex genetic variations. In SNP2Vec: Scalable Self-Supervised Pre-Training for Genome-Wide Association Study, Samuel Cahyawijaya et al. applied BPE to single nucleotide polymorphisms (SNPs). By grouping frequent genetic variations into variable-length subwords, the model captures complex, multi-marker biological motifs while remaining compact enough to process massive genome-wide association data within standard memory limits.

Expression-to-Text Transformation for Cellular Profiles

Tokenization strategies extend beyond linear sequences like DNA or proteins. Single-cell transcriptomics (scRNA-seq) presents a completely different data structure: a single cell is characterized by a tabular vector of continuous numerical values representing the expression levels of thousands of genes. These genes have no natural linear order.

To force this high-dimensional tabular data into a sequence-based Transformer, researchers developed rank-based tokenization, often called the expression-to-text transformation. Instead of feeding raw, continuous numbers into the model, individual genes are treated as tokens. For each cell, these gene tokens are sorted in descending order based on their expression intensity. The most highly active genes become the first "words" in a structured "cell sentence."

Raw scRNA-seq Matrix:   [GeneA: 0.2, GeneB: 9.4, GeneC: 1.5, GeneD: 0.0]
Rank-Sorted Sentence:   "GeneB  GeneC  GeneA"  (Zero-expression genes omitted)

Why does this work? Sorting genes by relative rank preserves the core biological signal while discarding technical noise. Cong Li et al. demonstrate in scReader: Prompting Large Language Models to Interpret scRNA-seq Data that rank-based sequences mitigate the severe batch effects and scaling variations inherent in raw numerical counts. Highly expressed genes remain at the front of the sentence across different sequencing runs, stabilizing the input.

By framing cellular states as sentences, models can leverage standard self-attention mechanisms to learn how genes interact. As explored by Chen Fang et al. in How do Large Language Models understand Genes and Cells and Yuhang Chen et al. in CellDuality: Unlocking Biological Reasoning in LLMs with Self-Supervised RLVR, ordering features by expression level allows the Transformer to focus on functional dependencies. The model learns the "grammar" of a cell by analyzing which gene tokens frequently appear together at the beginning of a sentence, mapping complex cellular identities into a format optimized for language-based reasoning.

Key Foundational Work

DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome (2020) — contributed 1 concept elements to this direction.
Exploring Chemical Space using Natural Language Processing Methodologies for Drug Discovery (2020) — contributed 1 concept elements to this direction.
Identification of Enzymatic Active Sites with Unsupervised Language Modeling (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_k_mer_tokenization_ddf179e6,
  title = {k-mer tokenization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/k-mer-tokenization/rdr_ddf179e69bb0412aacbf519c916e15c2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_k_mer_tokenization_ddf179e6_v1,
  title = {k-mer tokenization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/k-mer-tokenization/rdr_ddf179e69bb0412aacbf519c916e15c2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake