The Genomic Token Tax: Does a Reference-Built Vocabulary Tax Everyone Else's DNA?

DNA language models first cut the genome into "tokens," the discrete units the network reads. Most recent models learn that cut from data. Byte Pair Encoding (BPE) is a compression method that repeatedly glues together the most frequent adjacent letter pairs. It learns its vocabulary from whatever genomes it is shown, and those are usually one human reference assembly plus a pool dominated by well-studied species. In ordinary text, a tokenizer built mostly on English splits African languages into many more pieces per word, and this "token tax" explains up to half of the accuracy gap across languages. We ask whether genomic tokenizers impose the same tax on DNA the reference never saw: other human haplotypes (the sequence a person inherits on one chromosome copy), and species that are rare in the training pool.

The practical stakes are specific. Genomic models are already known to be nearly blind to single-letter mutations. Their embeddings of a reference sequence and a mutated one have cosine similarity above 0.99, and their zero-shot clinical variant scores sit at chance (Tokenization to Transfer). Our hypothesis is that part of this blindness comes from a vocabulary learned on a single source. In that vocabulary, sites where people actually differ get buried inside long merged tokens. We propose to measure the tax directly. We will then test one targeted fix: a BPE variant that is forbidden from merging across positions known to vary between people. This fix keeps most of BPE's compression, which character-level models give up.

Core Claim

The closest existing work already varies the tokenizer type (character, $k$-mer, BPE) and the pre-training corpus (human reference, 1000 Genomes, multi-species), so the "obvious" study is partly done (Tokenization to Transfer). What no paper in the file separates is the source the tokenizer's vocabulary is learned from, as distinct from the source the network is pre-trained on. We claim three things. First, the vocabulary source alone produces a measurable fertility gap, meaning more tokens per kilobase, on held-out haplotypes and on under-represented species. Second, that gap predicts downstream loss on those groups, the way fertility predicted accuracy in The Token Tax. Third, keeping common polymorphic sites as token boundaries recovers single-variant sensitivity at a small compression cost. If the tokenizer source has no effect once the pre-training corpus is held fixed, then the bias lives entirely in the corpus. That null result would also be informative, because it would tell practitioners that re-mixing data is enough and tokenizer engineering is not needed.

Research Program

The work relates to prior papers in three roles. Some supply the method. DNABERT-2 introduced BPE for DNA and showed roughly 5-fold compression over overlapping $k$-mers, along with gains on the 28-task Genome Understanding Evaluation (GUE). It trains on 32.49 billion bases from 135 species treated as a single pool and never asks which species the merges favour. We reuse its architecture and training recipe as our model backbone. MergeDNA learns token boundaries end-to-end with differentiable merging. It raises the same open question in sharper form: does a learned tokenizer merge by biological function, or by the redundancy of over-represented species? We include it as a comparison but do not build on it.

Other papers motivate the question. The Token Tax established the link between fertility and accuracy for human languages. AtlasNLP showed that a single curated repository hides who is actually represented in the data: 74% of its records lack a documented represented country. In genomics, the analogue is that the provenance of a reference assembly is rarely reflected in how models are evaluated.

One paper constrains the design. Tokenization to Transfer reports that pre-training helps only 2–3% over random initialization and that character-level models trained from scratch are strong. Any tokenizer effect we claim therefore has to survive a random-initialization control and a character-level control at matched compute.

The build proceeds in four steps. First, we learn BPE vocabularies of fixed size from three sources: the human reference only, the DNABERT-2 multi-species pool, and a haplotype-diverse human set drawn from 1000 Genomes. We then audit each vocabulary without training any model. We compute tokens per kilobase on held-out 1000 Genomes haplotypes and on species groups sorted by how much they contribute to the pool. We also count how often a common single-nucleotide variant falls strictly inside a token of length greater than one.

Second, we train a small grid of roughly 117M-parameter DNABERT-2-style encoders. The grid crosses tokenizer source with two pre-training corpora (human reference and multi-species), under identical token budgets measured in bases, not tokens. Measuring in bases keeps compression differences from silently changing how much DNA each model sees.

Third, we add the intervention, which we call variant-boundary BPE. Merge learning is run as usual, except that any candidate merge spanning a position with allele frequency above a threshold in the diverse panel is disallowed. A common variant then changes exactly one short token instead of re-segmenting a long one. We sweep the threshold to trace the full trade-off between compression and sensitivity.

Fourth, we fit a per-group regression of downstream score on fertility, following the Token Tax analysis, to test whether fertility carries predictive signal beyond pre-training corpus membership.

Baselines

The naive reference points are a randomly initialized model and a character-level model trained under the same budget. Tokenization to Transfer reports that a randomly initialized 8M-parameter Caduceus reaches an average MCC (Matthews correlation coefficient) of about 0.62 on hard histone and enhancer tasks, beating pre-trained NT-500M. Character tokens have no merge table and therefore cannot carry vocabulary bias, so they set the level of sensitivity our fix must reach. Both will be trained on our data with the same fine-tuning protocol and the same hyperparameter search budget.

The strongest existing static-tokenizer model is DNABERT-2 itself. Its reported result is an average improvement of 6 absolute points over DNABERT on GUE, winning 23 of 28 datasets. Our multi-species-tokenizer, multi-species-corpus cell reproduces it, so the comparison runs on identical code. The strongest dynamic tokenizer is MergeDNA, reported at 77.11% mean on GUE, 78.39 on the Nucleotide Transformer benchmark, and 90.87% on Genomic Benchmarks. We will fine-tune released checkpoints where available under our protocol rather than quote numbers from different splits.

For variant sensitivity, the bar is the ClinVar zero-shot log-likelihood-ratio AUROC of 0.345–0.536 reported across seven genomic models, together with embedding cosine similarity above 0.99. A model beats this bar if its AUROC is clearly above 0.55 and its cosine similarity for mutated sequences is clearly below 0.99.

Two ablations isolate the key ingredient. The first forces the same number of boundaries at random non-polymorphic positions, which separates the effect of lower compression from the effect of boundary placement. The second uses the diverse-panel vocabulary without the boundary constraint, which separates "more diverse merge statistics" from "boundaries at variants."

Variant-boundary BPE beats the field if it does three things together. It must match DNABERT-2's GUE average within noise. It must shrink the fertility gap between groups. And it must lift ClinVar AUROC toward the character-level model, while keeping at least three-quarters of BPE's compression.

Why not just…?

Why not just use character-level tokens and skip vocabulary bias altogether? This is the real threat. Character models avoid merge-table bias entirely, and Tokenization to Transfer shows they are strong from scratch. The same paper, however, found a 6-mer model outperforming a character model that had lower pre-training loss. Character inputs also give up roughly 5-fold compression, which matters under quadratic attention on long-range tasks. Our matched-compute comparison, run at fixed context measured in bases, settles the question directly. If character-level models match variant-boundary BPE on both GUE and ClinVar at equal cost, we will report that subword tokenization is not worth keeping for variant-facing work.

Why not just pre-train on a more diverse corpus and keep the standard tokenizer? The factorial grid tests exactly this. If the fertility gap and ClinVar results depend only on the pre-training corpus and not on the tokenizer source, then this alternative wins and our tokenizer claim fails.

Why not just re-learn BPE on the target population or species? This option is our diverse-panel-vocabulary ablation. It can reduce fertility, but it does not guarantee that polymorphic sites land on token boundaries. Common variants can still sit inside frequent merges, so we expect it to close the fertility gap without fixing mutation blindness. The ablation measures whether that expectation holds.

Proposed Experiments

Evaluation covers GUE (reported both overall and split by the source organism of each task), the Nucleotide Transformer benchmark, and Genomic Benchmarks, where the published baseline CNNs span 58.6–93.3% accuracy (Genomic benchmarks). Variant sensitivity is measured with ClinVar AUROC and with cosine similarity between reference and mutated sequences. We also measure fertility on held-out 1000 Genomes haplotypes and on low-share species. No reported result was found on Lacuna for GUE+ or the Long-Range Benchmark, so any numbers on those benchmarks will be our own runs.

Three checks decide the outcome. The first is whether vocabulary source changes fertility and downstream scores on under-represented groups when the pre-training corpus is held fixed. The second is whether per-group fertility explains a substantial share of score variance after controlling for corpus membership. The third is whether variant-boundary BPE beats the random-boundary ablation on ClinVar at equal compression. If the answer to the first is negative, the "genomic token tax" is a corpus problem, not a tokenizer problem. If the third fails, boundary placement is not the mechanism behind mutation blindness.