Do Compressed DNA Language Models Forget Non-European Genomes?

DNA language models now read windows of hundreds of thousands of bases. To make that affordable they compress the sequence. They drop or merge "uninformative" stretches of the input before attention, which is the step that compares every position to every other and so grows quadratically in cost. The compressors are trained and judged on reference genomes. A reference genome is a single consensus sequence with almost no human variation in it.

When these models are applied to a real person's genome, the informative content is that person's variants. People of African ancestry carry many more variants per window than people of European ancestry, and many of those variants are rare. This note proposes to test a simple, uncomfortable prediction. Under a fixed token budget, learned compression keeps a smaller share of variant-bearing positions in high-diversity genomes and distorts their positions more. If so, compression quietly widens the accuracy gap between ancestries, and the fix is mostly about where compression puts its budget and which coordinate system the surviving tokens keep.

The work joins two literatures that have not met. Vision and language work on token pruning has shown that surviving tokens must keep their original position labels, or the model's sense of distance and layout breaks. Statistical genetics has shown that tools built on European data fail elsewhere because the genome's local correlation structure differs across ancestries. We treat index-preserving compression as the mechanism and ancestry generalization as both the regime where it is stressed and the yardstick it is measured by.

Core Claim

Our claim is that compression-induced error in genomic transformers is ancestry-dependent, and that two specific design choices control most of the dependence. The first is whether the token budget is allocated in proportion to individual variant density or fixed per window. The second is whether surviving tokens keep reference-genome coordinates or are renumbered. Renumbering covers both packing survivors contiguously and using haplotype-local positions that shift with every insertion or deletion.

We expect a compressor with two properties to close most of the gap between compressed and full-context predictions, and to do so evenly across 1000 Genomes super-populations. It must anchor every non-reference allele and give positions in reference coordinates. Learned importance sampling of the kind used in Omni-DNA should not achieve this. If the gap does not depend on ancestry, the first half of the claim fails. If renumbering costs nothing once variant tokens are retained, the second half fails. Both outcomes are informative and both are cheap to observe.

Where the evidence comes from

The motivating evidence on the compression side is consistent. Keyformer prunes a language model's cached keys and values. It reports that renumbering the survivors damages accuracy, while keeping their original indices preserves 99–99.9% of baseline. Speculative Prefill makes the same point for prompt pruning, where keeping original position IDs allows up to 90% of tokens to be dropped. In vision, Nüwa shows that pruning by semantic importance alone collapses visual grounding to about 7% of full performance. Re-stretching positions over the original coordinate range recovers 47.2% at 64 tokens. Position Is All You Need (PAYN) goes further for dense segmentation. Uniform geometric sampling with original indices beats attention-scored pruning by 5–10% cIoU, because importance scores cluster and leave blind spots.

These papers motivate our design. PAYN's blind-spot finding also constrains it. A rare variant is exactly the kind of small, isolated feature that checkerboard sampling can skip. The file notes this "small-target obliteration" as an open problem. So we cannot simply import uniform sampling. Variant positions must be retained explicitly.

The closest existing system is Omni-DNA. Its SEQPACK operator applies position-aware learnable sampling to pack 450 kb windows into short sequences at 1.2–4.0% inference overhead, using rotary position embeddings to keep relative distances. Rotary embeddings encode position by rotating each token's vectors by an angle proportional to its index. Omni-DNA trains on RefSeq assemblies and never evaluates human variation, ancestry, or whether SEQPACK discards rare alleles. We reuse its setting and benchmark. We differ in asking what its compressor does to individual, ancestrally diverse haplotypes. A haplotype is the sequence of alleles a person inherits on one chromosome copy.

On the genetics side, three papers motivate the ancestry hypothesis and supply ground truth. TOP-LD uses TOPMed whole genomes to map ancestry-specific linkage disequilibrium, the correlation between nearby variants. It finds 137.8 million rare variants in the European panel alone and a 2.6–9.1-fold coverage gain over older panels. This is our source for per-ancestry allele frequencies and variant density. LOG-TRAM borrows European power into Biobank Japan only where local genetic correlation is high. It reports 1,954 lead variants, 842 of them novel, and this East Asian–discovered set gives us labeled positions to audit. Estimating Trans-Ancestry Genetic Correlation with Unbalanced Data Resources shows that ignoring LD differences biases cross-ancestry estimates by up to 5×. It warns that "same position, same meaning" is a European-centric assumption.

Research Program

We first build the evaluation, because none exists. We take 1000 Genomes phased haplotypes across the five super-populations. For each haplotype we write out its personal sequence over the Omni-DNA enhancer–target windows and over the shorter Nucleotide Transformer task regions, including SNVs and indels. A frozen DNA language model is run on each personal sequence twice. The first run uses full context, or a sliding-window full-attention reference where 450 kb is infeasible. The second run uses a compressor at a fixed retention budget of 10%, 25% and 50% of tokens.

The primary label-free metric is compression fidelity. This is the divergence between compressed and full-context outputs on the same haplotype. The second is variant-effect fidelity. This is the correlation, across variants, between the change in model output caused by an allele under compression and the same change under full context. The disparity metric is the spread of these fidelities across super-populations, reported after stratifying by per-window variant count. Stratifying lets us separate "African genomes have more variants" from "the compressor treats African variants worse at equal density."

Second, we decompose the error into three sources by controlled ablation. The first is token loss, meaning the variant-bearing token was dropped. The second is index distortion, meaning it survived but its position changed through packing or through upstream indels shifting haplotype-local positions. The third is attention renormalization, meaning the attention weights rescale once other tokens are gone, the effect Keyformer corrects with temperature.

Third, we build the proposed compressor. It is deliberately simple. Every token overlapping a non-reference allele is kept. The remaining budget is filled by farthest-point geometric sampling in the PAYN style. All survivors carry reference-genome coordinates, with insertions assigned fractional coordinates between their flanking reference positions. Allocation is per individual, so a window with more variants spends more of its budget on anchors. When anchors exceed the budget, we allocate the overflow using TOP-LD ancestry-matched LD, keeping one anchor per tightly correlated cluster.

Baselines

The naive reference point is the full-context model with no compression. This is the ceiling for fidelity by definition. On the labeled enhancer–target task, the strongest published number is the Omni-DNA paper's AUROC of 0.895, against 0.828 for HyenaDNA and 0.826 for Caduceus. That figure is measured on reference sequences. I found no reported result on Lacuna for any genomic compressor evaluated on personal or ancestry-stratified genomes, so the fidelity baselines are new measurements.

The strongest existing compressor is Omni-DNA's SEQPACK learnable sampling, run at the same retention budgets on the same backbone. If its released weights are unavailable, we reimplement it from the paper's description and verify it against the 0.895 reference figure before trusting any comparison. Beating it means two things at every budget: higher variant-effect fidelity in the African super-population, and a smaller fidelity spread across super-populations at matched variant density. Reference-sequence AUROC must not drop by more than a point.

Two general compressors are run under identical budgets. The first is PAYN-style uniform geometric sampling with preserved indices but no variant anchors, which tests whether geometry alone suffices. The second is average pooling over fixed strides, which Nüwa identifies as a strong layout-preserving baseline that blurs fine detail.

The key ablation keeps exactly the same retained tokens as our method but renumbers them contiguously or by haplotype-local position. This isolates the coordinate-frame ingredient. A second ablation keeps reference coordinates but drops the variant-density allocation. All methods share the backbone, sequences and budgets, and each learned component gets the same tuning budget on held-out chromosomes.

Why not just…?

Why not just skip compression and run full attention on shorter windows? This is the do-nothing answer, and for local tasks it may be right. The Nucleotide Transformer arms of our experiments will show whether compression is needed at all there. It does not cover long-range regulation. The enhancer–target benchmark exists because interactions span hundreds of kilobases, and short windows cannot see them. We report the full-context sliding-window reference everywhere, so the cost of not compressing is visible rather than assumed.

Why not just always keep the variant tokens and renumber freely? This is the real threat, since variant anchoring is the obvious heuristic and might deliver everything. The renumbering ablation above answers it directly. Keyformer and Nüwa both lead us to expect renumbering to hurt distance-dependent predictions such as enhancer–gene contact. Indels make it worse, because haplotype-local positions drift by an amount that varies by ancestry. If the ablation shows no loss, we will report that the coordinate frame does not matter in genomics. The contribution then shrinks to the audit and the allocation rule.

Why not just fine-tune the learned compressor on diverse human genomes? This may well reduce the disparity, and we include it as a baseline. We fine-tune SEQPACK-style sampling on 1000 Genomes haplotypes from all super-populations, holding out chromosomes, with the same compute as our method's tuning. It still learns importance from what is frequent in training, and PAYN's evidence is that importance scores leave blind spots on isolated features. The test is whether fine-tuning lifts fidelity for variants that are rare within each super-population. If fine-tuning matches our method there, the right recommendation is "train on diverse data," and we will say so.

Why not just use classical LD clumping to decide which positions to keep? Clumping picks the most significant variant per correlated block using reference LD. TOP-LD and the unbalanced-data paper both show that reference LD is European-skewed and miscalibrated elsewhere. We include ancestry-matched clumping, using TOP-LD panels, as the overflow rule. A European-LD clumping variant serves as a baseline, which directly tests the cost of the mismatch.

Proposed Experiments

The decisive first check is the audit itself. We run SEQPACK-style sampling at 25% retention on enhancer–target windows. We then plot variant-effect fidelity against per-window variant count, colored by super-population. If the fidelity curves coincide at matched density, the effect is purely density-driven and the allocation rule alone should fix it. If African haplotypes fall below the European curve at equal density, the compressor is biased by content rather than count.

The second check uses LOG-TRAM's lead variants. We measure how often a compressor retains, at its original coordinate, the token carrying a Biobank Japan–discovered novel lead variant versus a variant shared with UK Biobank, using East Asian 1000 Genomes haplotypes. Lower retention of the novel set would show that compression discards exactly the ancestry-specific signal that power-borrowing methods worked to find.

The third check is the full method comparison across budgets on the Nucleotide Transformer tasks and the enhancer–target task. We report reference AUROC, per-super-population fidelity, and the coordinate ablation.

The project succeeds in a narrow sense if it shows a measurable, density-adjusted ancestry gap in learned genomic compression. It succeeds fully if a variant-anchored, reference-coordinate compressor closes that gap without losing reference accuracy, while diversity fine-tuning alone does not.