Neutral Rewrites as a Contamination Probe for Genomic Foundation Models

Genomic foundation models — transformers trained on billions of DNA bases scraped from public reference databases — are now standard tools for predicting variant effects, gene expression, and pathogen identity. They are evaluated on benchmarks drawn from those same databases, which is like testing a student on sentences lifted verbatim from the textbook they memorized. In natural language processing this problem, benchmark data contamination, is well documented: models score high on static test sets because the answers leaked into pretraining. Genomics has almost no equivalent auditing, even though the evidence that leakage exists is already in hand: a mechanistic study of the Nucleotide Transformer found a cleanly isolated internal feature for the CMV viral enhancer in a model whose pretraining data was officially curated to exclude viral genomes, because the viral sequence had hitchhiked inside annotated bacterial reference entries Decode-gLM: Tools to Interpret, Audit, and Steer Genomic Language Models.

Biology offers something natural language does not: transformations of an input that provably preserve its meaning. Swapping a codon for a synonymous codon, or shuffling a regulatory region while preserving its short motifs, changes the surface string but not the biological function — the label stays valid by construction. This proposal turns that fact into a measurement instrument. Rewrite every benchmark sequence with a function-preserving "neutral paraphrase"; a model that has learned regulatory grammar should score identically, while a model that memorized exact reference sequences must degrade. Separately, the wet-lab world supplies a second instrument for free: sequencing technologies systematically decide which parts of life end up in databases at all — DNA-only pipelines discard RNA viruses, transcriptomics never sees silenced loci, amplicon assays drop mutated primer-binding sites. That bias is a natural exposure map over sequence space, and we can use it to stratify evaluation by how much of a test sequence the model plausibly saw during pretraining.

Core Claim

First, a function-preserving transformation operator applied to standard genomic benchmarks yields a quantitative memorization index — the paraphrase gap — that separates database recall from biophysical understanding. For a benchmark dataset $D$ of sequences with labels, and a neutral rewrite operator $T$ that alters the sequence while preserving the label by construction, the paraphrase gap is

$$\Delta_{\text{para}} = \mathrm{AUC}(D) - \mathrm{AUC}(T(D)),$$

where AUC is the area under the receiver operating characteristic curve for the same frozen model on the same task. A model relying on grammar should show $\Delta_{\text{para}} \approx 0$; a model relying on verbatim recall should show $\Delta_{\text{para}} > 0$, growing with rewrite intensity. Second, because sequencing bias controls the composition of reference databases, performance should split along bias-drawn lines — DNA versus RNA pathogens, transcriptionally active versus silenced regions, primer-conserved versus dropout loci — and those splits are contamination diagnostics, not biology.

The closest existing work motivates but does not perform this. The mechanistic audit in Decode-gLM found one contamination incident retrospectively, by matching internal features to pre-existing annotations; it supplies our sparse-autoencoder method but no general, quantitative evaluation metric. The paraphrase-laundering result in Test of Time: Rethinking Temporal Signal of Benchmark Contamination showed, for language models, that semantic rewrites hide memorization from overlap detectors — the influence-function hit rate for the true source document fell from 77.5% on exact cloze queries to 17.5% on rephrased ones. We invert that threat into a probe: in genomics, unlike in NLP, the paraphrase is function-certified, so the performance drop under rewriting is interpretable as memorization. Dynamic generators like DyVal and cryptographic verifiers like $\text{P}^4$Bench solve contamination for math and logic, where ground truth can be generated or verified symbolically; biology has no such engine, which is why transforming real data — rather than generating new tasks — is the right move here. Finally, the benchmarking suites themselves define the untouched surface: Benchmarking DNA foundation models for genomic and genetic tasks evaluates five models on 57 datasets without any contamination audit, and GenBench evaluates ten models on 43 static datasets the same way; neither asks whether the test sequences were in the training corpora.

Research Program

The program has four pieces. First, we build the neutral rewrite operator $T$ as a small library of task-matched transformations: codon-degenerate rewrites for coding tasks (replace codons with synonymous alternatives while keeping any variant allele fixed, so the pathogenic-versus-common label cannot change); motif-preserving local shuffles for regulatory tasks (permute non-coding segments while conserving transcription-factor motifs and dinucleotide statistics); and k-mer-statistics-preserving rewrites for taxonomic tasks. Every operator must satisfy a published neutrality contract — the label is preserved by the transformation's definition, not by hope.

Second, we build an exposure score. For each benchmark sequence we compute the fraction of its k-mers present in the model's pretraining corpus — the genomic analogue of the term-frequency analysis that Snoopy used to show few-shot performance tracks pretraining token frequency across more than 20 benchmarks. This is exact and enumerable for Gener anno, whose 715-billion-base-pair RefSeq pretraining corpus is public, making it the cleanest audit target; for models with unpublished corpora we estimate exposure from public database mirrors and say so.

Third, we stratify benchmarks along sequencing-bias axes drawn from the bias literature. The DNA/RNA divide means RNA viruses are systematically absent from DNA-optimized metagenomic databases: DNA-only clinical metagenomics excluded RNA respiratory viruses entirely Identifying subgroup of severe community-acquired pneumonia based on clinical metagenomics, a multicenter retrospective cohort study, and DNA-targeted assays in transplant and wildlife surveillance missed the RNA virome A diverse virome in kidney transplant patients contains multiple viral subtypes with distinct polymorphisms. Transcriptional capture means silenced loci never enter expression databases, mirroring the latent-provirus blind spot where near-full-length single-genome sequencing misses solo LTR structures and found 82% of defective pediatric proviruses carrying large internal deletions Differential HIV-1 Proviral Defects in Children vs. Adults on Antiretroviral Therapy. Amplicon dropout means the viral loci whose primer sites mutate during lineage evolution are under-represented, while molecular-inversion-probe tiling covered 99.69% of the SARS-CoV-2 genome A program for real-time surveillance of SARS-CoV-2 genetics. Each axis gives a predicted exposure gradient we can test against model performance.

Fourth, we add a mechanistic layer, reusing the sparse-autoencoder machinery of Decode-gLM to ask which internal features fire on original versus rewritten sequences, and influence functions in the style of Test of Time where training-data access permits. This is corroboration, not the headline: the headline metric must work on released checkpoints without training-data access.

Baselines

The do-nothing reference point is the untransformed benchmark evaluation itself: frozen model embeddings, mean token pooling, and a random-forest classifier, exactly the protocol that Benchmarking DNA foundation models established, in which mean pooling beat summary-token pooling by 1.4% to 8.7% AUC across 57 datasets and Nucleotide Transformer V2 reached an AUC of 0.73 on pathogenic-variant identification. Our pipeline must first reproduce that 0.73 within a stated tolerance on the original sequences before any transformation is applied; otherwise all downstream gaps are suspect. Beating this baseline is not the point — it is the calibration anchor — but the paraphrase gap is meaningful only relative to it.

The strongest existing competitor is sequence-identity decontamination: filter every benchmark sequence with high BLAST- or MMseqs2-style identity to the pretraining corpus, then evaluate on the survivors. This is standard bioinformatics practice and the natural first defense. We will run it exactly, on the same checkpoints, same frozen-embedding protocol, and same pooling, with the identical tuning budget (one nested validation split per task, no per-method hyperparameter fishing). For Gener anno the filter is exact because the RefSeq corpus is public. No reported result found on Lacuna for contamination-filtered genomic foundation model evaluation, so we must establish this baseline ourselves. It counts as beaten if the paraphrase gap on the filtered benchmark remains large — that is, if identity filtering fails to remove the memorization our probe detects. The literature predicts exactly this failure: CAMI II showed that alignment and binning break down above 95% average nucleotide identity among related strains Critical Assessment of Metagenome Interpretation: the second round of challenges, and the CMV enhancer hid inside entries whose metadata said "non-viral," defeating any identity or annotation filter Decode-gLM.

The strongest existing audit method is influence-function tracing, the EK-FAC/Kronfluence approach that identified the true source document as the most influential training point 77.5% of the time for exact cloze queries in Test of Time, adapted here to trace genomic predictions back to training contigs. It is the competitor because it is the only prior tool that makes memorization quantitative rather than anecdotal. We will run it wherever training data is enumerable (Gener anno; a small retrained HyenaDNA). It is beaten on two grounds we will demonstrate: it requires training-data and gradient access that released checkpoints do not grant, and it measures provenance (did this sequence influence the model) rather than reliance (does the model's correct answer depend on the exact string) — the question the paraphrase gap answers directly.

The key-ingredient ablation is a rewrite-intensity sweep plus a destructive control. Intensity is the fraction of rewritable positions actually rewritten, swept from 0 to the maximum the neutrality contract allows; a memorizing model should show a smooth monotonic AUC decay while a grammar model stays flat, so the shape of the curve carries the diagnosis, not any single point. The destructive control scrambles sequences in ways that break function while preserving length and composition; every model, however good, must collapse toward chance under it. A model that survives the destructive control but dies under the neutral rewrite is the signature of memorization; a protocol that cannot produce that contrast is invalid, and we would report it as such.

Why not just…?

Why not just deduplicate training and test sets with BLAST? This is the strongest simple alternative, and we treat it as the primary baseline rather than arguing around it. The evidence that it falls short is threefold from the file: identity thresholds are unreliable in dense homology space, as CAMI II's strain-resolution failure above 95% ANI shows Critical Assessment of Metagenome Interpretation; paraphrase-style surface change launders memorization from exact-match detection, as the influence hit rate collapse from 77.5% to 17.5% under rephrasing demonstrates Test of Time; and contaminants hide inside entries whose species-level metadata gives no signal, as the CMV-enhancer case proved Decode-gLM. Synonymous codon swaps preserve the memorized motif content while cutting global identity — precisely the laundering geometry — so dedup by identity should miss what our probe catches, and our filtered-baseline experiment is designed to show that concretely rather than assert it.

Why not just use a time split — train on data deposited before a date, test on what came after? Temporal holdout is fragile in language Test of Time, where post-cutoff performance on rephrased physics questions actually rose 2.19 percentage points, and it is worse in biology because evolution barely generates novelty: emerging SARS-CoV-2 lineages shared overwhelming backbone homology with pre-cutoff lineages A program for real-time surveillance of SARS-CoV-2 genetics, so a model pretrained before 2021 already contains nearly the whole genetic background of a 2023 variant. A post-2022 genome test set is not an uncontaminated test set; it is the same sequences with a few mutations, which is functionally the same situation our neutral rewrite creates deliberately and with a label guarantee. We will include a date-based split on RefSeq entries as a comparison axis, and expect it to look reassuring while the paraphrase gap persists — that contrast is itself a result.

Why not just evaluate with zero-shot frozen embeddings, which already avoids fine-tuning leakage? Zero-shot embedding evaluation prevents leaking the downstream task labels, not the pretraining sequences; the CMV feature was found in a model analyzed with no fine-tuning at all Decode-gLM. It is also an unstable measurement in itself: switching only the pooling rule moved AUC by 1.4% to 8.7% across the 57-dataset suite Benchmarking DNA foundation models, so any contamination signal smaller than that would be unmeasurable without a fixed protocol. We adopt frozen embeddings with mean pooling as our evaluation substrate precisely for this reason — it is the fairest existing protocol — and then apply the rewrite on top of it, so the comparison is protocol-matched and the only varying ingredient is the sequence surface.

Proposed Experiments and Decisive Checks

Experiment one runs the paraphrase gap across the 57-dataset suite Benchmarking DNA foundation models and selected GenBench tasks GenBench for five checkpoints (DNABERT-2, Nucleotide Transformer V2, HyenaDNA, Caduceus-Ph, GROVER), anchored on the reproduced 0.73 pathogenic-variant AUC. Experiment two performs exact contamination accounting on Gener anno Gener anno, correlating per-sequence exposure scores with per-sequence paraphrase gaps on the Prokaryotic Gener Tasks; the prediction is that high-exposure sequences show the largest gaps. Experiment three stratifies along the bias axes: DNA-versus-RNA viral classification, expressed-versus-silenced regions, and primer-conserved-versus-dropout loci, where we predict performance tracks database presence rather than biological difficulty. Experiment four is the causal check: retrain a small HyenaDNA — feasible, since the same architecture was retrained on 135 species Benchmarking DNA foundation models — on deduplicated versus unmodified corpora, with the prediction that the deduplicated model shows a near-zero paraphrase gap. Experiment five applies sparse-autoencoder tracing Decode-gLM to confirm that predictions on original sequences route through database-identity features that go silent under rewriting.

The decisive outcomes are symmetric and both publishable. If paraphrase gaps are near zero across models and intensities while the destructive control collapses to chance, then current genomic foundation models genuinely learned grammar rather than reference-database recall, and the field's headline numbers — like the 0.73 AUC — survive a contamination audit for the first time. If gaps are large, grow with rewrite intensity, correlate with exposure, and shrink after deduplicated retraining, then a substantial share of reported genomic model performance is memorization, and the paraphrase gap becomes the field's standing contamination metric, the thing CAMI II's finding that only 3 of 10 clinical-pathogen submissions identified the causal virus Critical Assessment of Metagenome Interpretation already hints the community needs. The honest risk is that our rewrites are not perfectly neutral — synonymous codons can shift translation efficiency — which would inflate gaps and fake a memorization verdict; the destructive control, the intensity sweep, and a comparison against a specialized biology-respecting predictor of the Enformer type on the same rewritten inputs are the guardrails, and if the specialized predictor degrades equally, we report the transformation as the culprit rather than the model.