Does DNA Redundancy Become Parameter Redundancy?

Pruning results from language models suggest that bigger networks carry more spare capacity. One-shot pruning can delete half the weights of a 175-billion-parameter model with little loss, and a pruned large model usually beats a dense small one of equal size. Genomic sequence models are trained on data that is far more repetitive than text: two human genomes are about 99.9% identical, and much of each genome is repeats. This proposal asks whether that data redundancy reappears as parameter redundancy inside the model. If it does, how prunable a DNA model is should depend on the training corpus as well as on parameter count, and larger DNA models should be wasteful in a way that can be measured and removed.

The question matters for a practical reason and for a scientific one. In practice, genomic foundation models are scaling fast but behave oddly at the top end. A 40-billion-parameter Evo2 model collapses to an AUC of 0.48 on standard GPUs, while the 7B model stays stable, and a 265M model with biologically informed tokens beats much larger models. Scientifically, genomics lets us set the redundancy of the training data precisely. We can add more individuals, mask repeats, or mix species, all at a fixed token budget. Text does not allow controls this clean, so a "redundancy scales with data" claim can be tested here directly.

Core Claim

Our hypothesis is that a model's spare capacity has two sources. One is parameter count, which the language-model literature documents. The other is the compressibility of the training corpus, which nobody has measured. We will measure the critical sparsity $s^*(N, R)$. This is the largest fraction of weights that one-shot pruning can remove while held-out loss stays within $\delta = 0.01$ nats per base of the dense model. Here $N$ is parameter count and $R$ is corpus redundancy, measured as one minus the bits-per-base a strong universal compressor achieves divided by 2. The claim is that, at fixed $N$ and fixed training tokens, $s^*$ rises with $R$. We also claim that the gain from "train large, then prune" over "train small and dense" is larger for genomic models than the language-model literature reports.

If this holds, three consequences follow. Scaling decisions for DNA models should be made against the non-redundant information in the corpus rather than raw token count. Existing large DNA models can be shrunk substantially without retraining. And deduplicating the corpus before training becomes a competing explanation that has to be tested directly, which we do.

Relation to Prior Work

Several papers motivate the hypothesis on the model side. Universal Properties of Activation Sparsity in Modern Large Language Models shows that activation redundancy grows and effective activation rank falls with language-model scale. SparseGPT prunes a 175B model by 50–60% in one shot. Enhancing One-Shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism shows the same procedure breaks small models. Two Sparse Matrices are Better than One and To Prune, or Not to Prune show that pruned large models beat dense small ones. All of these vary only the model and treat the data as fixed. We vary the data.

Two papers supply the measurement tools. Effective Model Pruning estimates how many components actually carry importance, using an effective-count index borrowed from ecology. Iterative Magnitude Pruning Reduces Weight-Space Coupling defines LOGDET, a Fisher-based score of how strongly weights interact, and reports that it rises again below about 10% density. That reversal also limits how far we expect pruning to go. Maximum Redundancy Pruning provides a layer-wise sparsity allocation that we use as a strong pruner rather than as an object of study.

On the genomic side, existing work compresses along other axes, never the weights. GeneZip compresses sequence length by routing repetitive DNA into longer tokens. A Benchmark of Evo2 Genomic AI Models studies FP8 quantization and reports the 40B collapse without explaining it. Mamba-Shedder prunes state space models (recurrent sequence models with a learned linear state update) but only on language benchmarks. BioToken and BioFM chooses a small model size up front and adds annotation tokens. Two classical results act as a constraint and a reality check. Genomic Data Classification via Universal Compression shows an LZ78 dictionary classifier matching DNABERT-2 on 89.2% of GUE datasets, but lagging by 13–24% on promoters and splice sites. We reuse its compression log-loss as our corpus-redundancy measure $R$. Ultrafast classical phylogenetic method beats large protein language models suggests that much neural capacity in biology may be superfluous. Our proposal turns that suspicion into a measurement.

Research Program

The core of the program is a controlled grid of models we train ourselves. We use two backbones. One follows Caduceus (Caduceus), a bidirectional state space model built so that its output on a DNA strand matches its output on the reverse-complement strand. The other follows HyenaDNA (HyenaDNA), which uses long convolutions computed with fast Fourier transforms. Each backbone is trained at five sizes from roughly 2M to 500M parameters.

Each size is trained on four corpora with an identical token budget:

  • the single human reference genome;
  • the reference plus many 1000 Genomes haplotypes, which is highly redundant;
  • the repeat-masked reference, which is less redundant;
  • a multi-species corpus, which is least redundant.

For each corpus we measure $R$ with LZ78 sequential log-loss and with a second compressor as a robustness check.

For every trained model we record four quantities at checkpoints through training. The first is $s^*$ under SparseGPT-style one-shot pruning and under MRP allocation. The second is the effective rank of activations. The third is EMP's effective component count. The fourth is LOGDET, computed only for models up to about 50M parameters, where an approximate Fisher matrix is tractable.

Pruning the Caduceus models raises a design problem. Unconstrained pruning breaks the reverse-complement symmetry. We therefore implement a tied variant that removes paired weights together, and we report both variants.

We then move to public models. We apply the same measurements to HyenaDNA checkpoints and to Evo2 at 1B, 7B and 40B parameters. The goal is to test whether the 40B model's numerical collapse coincides with large activation outliers or re-coupled weights. We treat this as an exploratory diagnosis, not a promised explanation.

The practical deliverable is a test of "prune from large" in genomics. Evo2-7B is pruned to the non-zero parameter count of Evo2-1B and compared against dense Evo2-1B on the same tasks.

Baselines

The naive reference point is the dense model at each size and corpus, evaluated with no pruning. It defines the $\delta$ threshold, and every $s^*$ is measured against it. A second naive point is uniform magnitude pruning at fixed sparsity. Maximum Redundancy Pruning reports that uniform pruning gives LLaMA2-7B a perplexity of 80.40 at 70% sparsity, against 23.54 with MRP and 30.47 with OWL. That gap is why uniform pruning would understate redundancy, so we use MRP and SparseGPT-style pruning as the primary pruners. All pruners receive the same 128-sequence calibration set drawn from held-out chromosomes and the same compute.

The strongest existing competitor for small genomic models is a compact, biologically informed model trained from scratch. BioFM reports an overall Variant Benchmark AUROC of 0.780, against 0.727 for the second-best model, NT-500M-v2-MS (BioToken and BioFM). A pruned large model beats this baseline only if it exceeds 0.780 at equal or smaller non-zero parameter count without using annotation tokens, or matches it on an unannotated species where BioFM cannot be built.

For the Evo2 comparison, the reference is the deployment benchmark's dense 7B model, which reaches AUC 0.88–0.95 on the cancer SNV pathogenicity sets under FP8 on H800 (A Benchmark of Evo2 Genomic AI Models). Pruned Evo2-7B beats dense Evo2-1B if it scores higher on all three cancer datasets at the same non-zero count. It is competitive with the 7B model if it stays inside that 0.88–0.95 band. For Caduceus and HyenaDNA on genomic-benchmarks, the file lists only the original CNN baselines, for example 93.3% on human_ensembl_regulatory (Genomic benchmarks). We found no reported Caduceus or HyenaDNA accuracy on Lacuna, so we will rerun both.

The ablation that isolates our key ingredient holds $N$, architecture and tokens fixed and varies only the corpus. If $s^*$ does not move with $R$, the data-to-parameter claim fails, regardless of how well pruning works.

Why not just…?

Why not just deduplicate the corpus and train a small model? This is the most serious threat. If spare capacity comes from data redundancy, removing that redundancy and training small could make pruning unnecessary. We test it directly. For each size, a model trained on a deduplicated, repeat-masked corpus at matched compute is compared with a larger model trained on the redundant corpus and then pruned to the same size. If deduplication matches prune-from-large, we report that as the main practical finding. The scientific claim about $s^*$ and $R$ survives either way.

Why not just build compact models with biological priors, as BioFM and Caduceus do? Those models are strong, and BioFM's 0.780 AUROC is our bar. But BioToken depends on GENCODE annotations, which do not exist for most species. Prior work also finds that winning tickets carved from large models have decoupled weight geometry that equal-size dense models do not reach (Iterative Magnitude Pruning Reduces Weight-Space Coupling). Our evaluation on the multi-species arm and on an unannotated held-out species tells us whether priors or pruning generalize better.

Why not just quantize to FP8? FP8 helps a lot: the Evo2 benchmark reports going from 20,024 MiB on A100 to 3,156 MiB on H800 for the 7B model. But it requires specific hardware, and it does not stop the 40B collapse on A100. It is also orthogonal to pruning, so we report pruning both alone and combined with quantization, at matched memory.

Why not just use LZ78 or other classical methods? On local tasks such as epigenetic marks, LZ78 classification may well win, and we include it on GUE. The same paper reports that it lags by 13–24% on promoters and splice sites, and those long-range tasks are where we evaluate pruned neural models. Using the compressor's log-loss as $R$ also makes the classical view part of the measurement rather than a competitor.

Proposed Experiments and Decisive Checks

The first decisive check is the controlled grid. We regress $s^*$ on $\log N$ and $R$ with backbone fixed effects. The claim survives only if the coefficient on $R$ is positive with a bootstrap interval excluding zero, in both the Caduceus-style and HyenaDNA-style backbones. A null result here ends the central hypothesis.

The second check asks whether the prune-from-large advantage is real for DNA. At each size we compare held-out loss for a pruned larger model, a dense model of matching size, and a model trained on the deduplicated corpus. We do this on genomic-benchmarks, GUE and DNALongBench, where GeneZip reports a 50.4x fine-tuning speedup on eQTL that we treat as an efficiency reference.

The third check is on public models. Evo2-7B is pruned to 1B non-zero parameters and evaluated on the three cancer SNV datasets and the Variant Benchmark against the numbers above.

The fourth check concerns reverse-complement symmetry. We measure equivariance error and accuracy for tied versus untied pruning of Caduceus, to show whether respecting the DNA symmetry costs capacity or saves it.

The fifth check is exploratory. We test whether Evo2-40B's activation outliers and LOGDET-style coupling, measured on sampled layers, are anomalous relative to the 7B model. That would connect its hardware fragility to over-redundancy, though we commit only to reporting the measurement.