dnaGrinder: a lightweight and high-capacity genomic foundation model

Qihang Zhao, Chi Zhang, Weixiong ZhangSubmission to ICLR 2025 · Withdrawn

TL;DR

dnaGrinder is a lightweight, encoder-only genomic foundation model (63.6M parameters) that handles long DNA sequences while slashing computational and memory requirements. By combining a memory-efficient Byte Pair Encoding (BPE) tokenizer, Sequence Length Warmup, Attention with Linear Biases (ALiBi), and FlashAttention-2, dnaGrinder outperforms or matches models up to 40× its size (such as NT-2500M) while requiring only ~69.5 billion pretraining tokens.

What Problem Does This Paper Address?

Genomic sequences feature complex, long-range dependencies and vast nucleotide variations. Existing genomic foundation models force researchers to choose between two extremes:

  1. Compact models that struggle with accuracy or sequence context windows (often capped at 128 to 512 tokens).
  2. Massive models (e.g., Nucleotide Transformer with 2.5B parameters) that demand prohibitive computational resources for pretraining and fine-tuning.

Additionally, standard BPE tokenizers consume terabytes of RAM when training on multi-gigabyte genomic corpora, and most models learn from redundant, repeat-heavy sequences or miss structural genetic variations.

Methods

dnaGrinder introduces several engineering and algorithmic choices within an encoder-only architecture:

  • Memory-Efficient BPE (ME-BPE): Standard BPE over huge FASTA corpora exceeds workstation memory limits (e.g., >1.8 TB RAM for a 118 GB corpus). ME-BPE streams data iteratively across smaller file splits, updating and pruning token frequencies toward a target vocabulary of 4,096 tokens (~5 base pairs per token).
  • Sequence Length Warmup (SLW): dnaGrinder is the first encoder-based model to use SLW. Pretraining batches are organized in ascending order of tokenized sequence lengths. This allows the model to learn simple, low-complexity motifs first before tackling complex, entropy-dense sequences.
  • ALiBi & FlashAttention-2: Instead of Rotary Positional Embeddings (RoPE), which decay over long genomic distances, dnaGrinder uses Attention with Linear Biases (ALiBi) to extrapolate to context windows 10× longer than those seen during pretraining. FlashAttention-2 cuts memory footprint and accelerates attention runtime.
  • SwiGLU Activation: Replacing standard GELU/GEGLU with SwiGLU keeps parameter size low (63.6M vs. 110M for GEGLU) without sacrificing capacity.

Datasets

  • Pretraining Datasets:
  • Human Reference Genome (GRCh38.p14): Repetitive regions were soft-masked and filtered to prioritize informative, non-redundant genomic sequences extended to 12,200 bp.
  • 1000 Genomes Project: Incorporates Single Nucleotide Variants (SNVs), INDELs, and Structural Variants (SVs > 50 bp) using both maternal and paternal phased alleles.
  • Multispecies Corpus: Reference genomes spanning 794 species across diverse kingdoms from NCBI.
  • Downstream Benchmarks:
  • GUE Benchmark (28 tasks): Includes promoter and core promoter detection, splice site detection, human/mouse transcription factor binding, virus variant classification, and yeast epigenetic marks.
  • Enhancer Tasks (2 tasks): Enhancer identification and enhancer type classification from Nucleotide Transformer (NT).
  • Long-Sequence Species Classification: 120,000 bp classification task across 5 mammalian species (hippo, human, lemur, mouse, pig).

Key Insights and Ideas

  • Curriculum through Compression: Token length reflects biological complexity. Repetitive sequences compress into fewer tokens, while highly variable regions require more tokens. Sorting training data by token count naturally creates a simple-to-complex learning curriculum via SLW.
  • Phasing and Structural Variants Matter: Unlike prior work that only incorporates maternal or paternal single-base variations, incorporating phased dual-parental variants alongside large structural variants (>50 bp) significantly enriches sequence diversity without duplicating reference backbones.
  • Extrapolation Outperforms Scaling: Massive parameter counts are not strictly necessary for genome modeling. Combining ALiBi with a 63.6M model enables zero-shot length extrapolation up to 120,000 bp during inference.

Key Results

  • Benchmark Superiority: Across 30 downstream tasks (GUE + Enhancer benchmarks), dnaGrinder achieved the highest average evaluation score (73.01) compared to DNABERT-2 (70.86) and the 40× larger NT-2500M-multi (68.32). It ranked 1st in 11 tasks and 2nd in 12 tasks.
  • FLOPs & Data Efficiency: dnaGrinder trained on only 69B tokens, requiring ~29× fewer FLOPs than NT-2500M (300B tokens) and 1.8× fewer FLOPs than DNABERT-2 (262B tokens).
  • Extreme Sequence Handling: On a single GPU, only dnaGrinder and HyenaDNA could handle 120,000 bp inputs; dnaGrinder achieved 100% accuracy on species classification, whereas HyenaDNA scored 64.22% (DNABERT-2 and NT ran out of memory).
  • Accessible Footprint: dnaGrinder can fine-tune sequences exceeding 17,000 tokens on a consumer 12 GB GPU (RTX 4070) and over 140,000 tokens on an 80 GB GPU (H100/A800).

Limitations and Future Work

  • Hyperparameter Sensitivity: The model displays sensitivity to learning rate variations during fine-tuning (shifts as small as $0.1 \times 10^{-5}$ altered downstream performance), requiring fine-grained sweeps and multiple seed runs.
  • Mixed Utility of Further Pretraining: In-domain continued pretraining yielded mixed results—improving accuracy on 4 yeast epigenetic tasks but degrading performance on 6 others, indicating limited general utility.
  • Difficult Tasks: All baseline models, including dnaGrinder, struggled to converge consistently on certain noisy downstream tasks, such as COVID-19 viral variant classification.

Summary updated

Abstract

The task of understanding and interpreting the complex information encoded within genomic sequences remains a grand challenge in biological research and clinical applications. In this context, recent advancements in large language model research have led to the development of both encoder-only and decoder-only foundation models designed to decode intricate information in DNA sequences. However, several issues persist, particularly regarding the efficient management of long-range dependencies inherent in genomic sequences, the effective representation of nucleotide variations, and the considerable computational costs associated with large model architectures and extensive pretraining datasets. Current genomic foundation models often face a critical tradeoff: smaller models with mediocre performance versus larger models with improved performance. To address these challenges, we introduce dnaGrinder, a unique and efficient genomic foundation model. dnaGrinder excels at managing long-range dependencies within genomic sequences while minimizing computational costs without compromising performance. It achieves results that are not just comparable but often superior to leading DNA models such as Nucleotide Transformer and DNABERT-2. Furthermore, dnaGrinder is designed for easy fine-tuning on workstation-grade GPUs, accommodating input lengths exceeding 17,000 tokens. On a single high-performance GPU, it supports sequences longer than 140,000 tokens, making it a highly efficient and accessible tool for both basic biological research and clinical applications.
BibTeX
@misc{zhao2024dnagrinder,
  author = {Qihang Zhao and Chi ZHANG and Weixiong Zhang},
  title = {{dnaGrinder: a lightweight and high-capacity genomic foundation model}},
  year = {2024},
  eprint = {2409.15697},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.15697},
}
Figure