TL;DR
dnaGrinder is a lightweight, encoder-only genomic foundation model (63.6M parameters) that handles long DNA sequences while slashing computational and memory requirements. By combining a memory-efficient Byte Pair Encoding (BPE) tokenizer, Sequence Length Warmup, Attention with Linear Biases (ALiBi), and FlashAttention-2, dnaGrinder outperforms or matches models up to 40× its size (such as NT-2500M) while requiring only ~69.5 billion pretraining tokens.
What Problem Does This Paper Address?
Genomic sequences feature complex, long-range dependencies and vast nucleotide variations. Existing genomic foundation models force researchers to choose between two extremes:
- Compact models that struggle with accuracy or sequence context windows (often capped at 128 to 512 tokens).
- Massive models (e.g., Nucleotide Transformer with 2.5B parameters) that demand prohibitive computational resources for pretraining and fine-tuning.
Additionally, standard BPE tokenizers consume terabytes of RAM when training on multi-gigabyte genomic corpora, and most models learn from redundant, repeat-heavy sequences or miss structural genetic variations.
Methods
dnaGrinder introduces several engineering and algorithmic choices within an encoder-only architecture:
- Memory-Efficient BPE (ME-BPE): Standard BPE over huge FASTA corpora exceeds workstation memory limits (e.g., >1.8 TB RAM for a 118 GB corpus). ME-BPE streams data iteratively across smaller file splits, updating and pruning token frequencies toward a target vocabulary of 4,096 tokens (~5 base pairs per token).
- Sequence Length Warmup (SLW): dnaGrinder is the first encoder-based model to use SLW. Pretraining batches are organized in ascending order of tokenized sequence lengths. This allows the model to learn simple, low-complexity motifs first before tackling complex, entropy-dense sequences.
- ALiBi & FlashAttention-2: Instead of Rotary Positional Embeddings (RoPE), which decay over long genomic distances, dnaGrinder uses Attention with Linear Biases (ALiBi) to extrapolate to context windows 10× longer than those seen during pretraining. FlashAttention-2 cuts memory footprint and accelerates attention runtime.
- SwiGLU Activation: Replacing standard GELU/GEGLU with SwiGLU keeps parameter size low (63.6M vs. 110M for GEGLU) without sacrificing capacity.
Datasets
- Pretraining Datasets:
- Human Reference Genome (GRCh38.p14): Repetitive regions were soft-masked and filtered to prioritize informative, non-redundant genomic sequences extended to 12,200 bp.
- 1000 Genomes Project: Incorporates Single Nucleotide Variants (SNVs), INDELs, and Structural Variants (SVs > 50 bp) using both maternal and paternal phased alleles.
- Multispecies Corpus: Reference genomes spanning 794 species across diverse kingdoms from NCBI.
- Downstream Benchmarks:
- GUE Benchmark (28 tasks): Includes promoter and core promoter detection, splice site detection, human/mouse transcription factor binding, virus variant classification, and yeast epigenetic marks.
- Enhancer Tasks (2 tasks): Enhancer identification and enhancer type classification from Nucleotide Transformer (NT).
- Long-Sequence Species Classification: 120,000 bp classification task across 5 mammalian species (hippo, human, lemur, mouse, pig).
Key Insights and Ideas
- Curriculum through Compression: Token length reflects biological complexity. Repetitive sequences compress into fewer tokens, while highly variable regions require more tokens. Sorting training data by token count naturally creates a simple-to-complex learning curriculum via SLW.
- Phasing and Structural Variants Matter: Unlike prior work that only incorporates maternal or paternal single-base variations, incorporating phased dual-parental variants alongside large structural variants (>50 bp) significantly enriches sequence diversity without duplicating reference backbones.
- Extrapolation Outperforms Scaling: Massive parameter counts are not strictly necessary for genome modeling. Combining ALiBi with a 63.6M model enables zero-shot length extrapolation up to 120,000 bp during inference.
Key Results
- Benchmark Superiority: Across 30 downstream tasks (GUE + Enhancer benchmarks), dnaGrinder achieved the highest average evaluation score (73.01) compared to DNABERT-2 (70.86) and the 40× larger NT-2500M-multi (68.32). It ranked 1st in 11 tasks and 2nd in 12 tasks.
- FLOPs & Data Efficiency: dnaGrinder trained on only 69B tokens, requiring ~29× fewer FLOPs than NT-2500M (300B tokens) and 1.8× fewer FLOPs than DNABERT-2 (262B tokens).
- Extreme Sequence Handling: On a single GPU, only dnaGrinder and HyenaDNA could handle 120,000 bp inputs; dnaGrinder achieved 100% accuracy on species classification, whereas HyenaDNA scored 64.22% (DNABERT-2 and NT ran out of memory).
- Accessible Footprint: dnaGrinder can fine-tune sequences exceeding 17,000 tokens on a consumer 12 GB GPU (RTX 4070) and over 140,000 tokens on an 80 GB GPU (H100/A800).
Limitations and Future Work
- Hyperparameter Sensitivity: The model displays sensitivity to learning rate variations during fine-tuning (shifts as small as $0.1 \times 10^{-5}$ altered downstream performance), requiring fine-grained sweeps and multiple seed runs.
- Mixed Utility of Further Pretraining: In-domain continued pretraining yielded mixed results—improving accuracy on 4 yeast epigenetic tasks but degrading performance on 6 others, indicating limited general utility.
- Difficult Tasks: All baseline models, including dnaGrinder, struggled to converge consistently on certain noisy downstream tasks, such as COVID-19 viral variant classification.
Summary updated