When a DNA language model judges whether a single-letter mutation is harmful, it asks one question: how surprised am I by this base at this position? Asking the same model the same question many times does little good, because its mistakes come from its weights and repeat on every query. This proposal tests a different way to get many opinions. We retrieve related stretches of DNA from other species, the evolutionary relatives of the query region, from an external database. We condition the model on each relative separately, so that each one yields its own prediction at the mutated base, and we combine those predictions by consensus. The bet is that votes conditioned on different relatives make mistakes that are much less correlated than votes from resampling. If that holds, consensus over retrieved context should keep improving as more relatives are added, while resampling the model stops helping almost immediately.
The idea connects three lines of work. Single-nucleotide tokenization, meaning one token per DNA letter, is what makes the vote well defined: every conditioned run returns a four-way distribution over A, C, G, T at the same query coordinate. A k-mer or byte-pair tokenizer, which groups letters into multi-letter tokens, offers no such shared coordinate. Self-consistency supplies the aggregation rule and the use of agreement as a confidence signal. External memory supplies the source of decorrelation. The main deliverable is a measurement, not just a score: how correlated per-vote errors are under memory-based voting versus sampling-based voting, and whether that correlation predicts the gains.
Core Claim
Self-consistency samples several reasoning paths and takes a majority vote over their answers. It works when wrong paths scatter across many different answers while correct ones converge, the intuition stated in Solving Quantitative Reasoning Problems with Language Models and surveyed in Towards Reasoning in Large Language Models: A Survey. The same literature also constrains the method. When errors come from a shared bias, voting amplifies them (BiasTrace). Unguided test-time sampling also saturates unless something external checks it (Scaling Test-Time Compute Without Verification or RL is Suboptimal).
The averaging argument makes the issue precise. Suppose each of $K$ votes has error variance $\sigma^2$ and any two votes have error correlation $\rho$. Then the averaged vote has error variance
$$\sigma^2\left(\rho + \frac{1-\rho}{K}\right),$$
which follows from summing pairwise covariances. Its floor is $\rho\sigma^2$, however large $K$ becomes. Our claim is that, for single-nucleotide variant scoring, conditioning on distinct retrieved homologs drives $\rho$ well below what sampling-style perturbations achieve. Here a homolog is a DNA segment in another genome that descends from the same ancestral sequence, and the perturbations are window shifts, reverse complements, or dropout. The result should be consensus gains that keep growing with $K$.
Concretely, consider a variant $v$ that changes the reference base $b_{\text{ref}}$ to the alternate base $b_{\text{alt}}$ in a query window $x$. For each retrieved homolog $r_k$, the score is
$$s(v) = \frac{1}{K}\sum_{k=1}^{K} w_k\left[\log p(x^{\text{alt}} \mid r_k) - \log p(x^{\text{ref}} \mid r_k)\right].$$
In this expression, $x^{\text{ref}}$ and $x^{\text{alt}}$ are the query window carrying each allele, $p(\cdot \mid r_k)$ is the model's likelihood of the window with $r_k$ placed in its context, and $w_k$ is a retrieval-similarity weight. Next to this averaged score we report a plurality vote: each homolog-conditioned run votes for whichever of the two alleles it prefers. The vote margin serves as a confidence score.
Relation to Prior Work
The closest work is E1: Retrieval-Augmented Protein Encoder Models, which motivates this proposal. E1 retrieves unaligned protein homologs and encodes them jointly with the query, and with retrieval its 600M model beats a model five times larger. E1 differs from our plan in three ways. It works on proteins rather than DNA, it trains a joint-attention model, and it fuses all homologs into one pass rather than treating them as separate votes. Whether joint fusion or separate voting is better is one of our central tests.
Fusion-in-Decoder supplies part of the method. It encodes each retrieved passage separately and fuses them only at the decoder. We go one step further and keep the passages separate all the way to the output, so that their disagreement stays measurable.
kNN-LM mixes the model's next-token distribution with one built from nearest neighbors in a datastore. In DNA its vocabulary would be just four bases. It is the strongest memory alternative that needs no in-context ability, and we run it as a baseline.
Three single-nucleotide models define the setting. HyenaDNA processes contexts of up to a million bases. SegmentNT annotates the genome base by base. JanusDNA is a bidirectional model at the same resolution. All three keep their knowledge in fixed weights and use no retrieval.
Two papers constrain the design. BMFM-DNA stores population variation inside the model by adding variant tokens to its vocabulary, which freezes a single snapshot of variation. We keep variation external instead. Lost in Tokenization reports that raw sequence hurts large language models doing protein question answering compared with text summaries. That result warns that retrieved raw DNA could act as noise. We address it directly by comparing joint concatenation with separate voting, and we do not assume retrieval helps.
Research Program
The datastore holds the multispecies corpora in the file: the Nucleotide Transformer 850-genome collection (The Nucleotide Transformer) and the DNABERT-2 135-species set (DNABERT-2). We split them into overlapping windows of about 1–2 kilobases and index each window in two ways: by minimizer seeds (short anchoring substrings used for fast alignment-free search) and by mean-pooled model embeddings. We use mean pooling because Benchmarking DNA foundation models found it beats summary-token pooling by 1.4–8.7% AUC. For every query, we remove the query's own human locus, and its human paralogs above an identity threshold, from the retrieval results.
Population data from the 1000 Genomes Project is left out of the main experiments. Common alleles are benign almost by definition, so a population memory would leak label information. We report it only as a separately marked condition.
The model is HyenaDNA. HyenaDNA needed soft prompting before it showed in-context learning, so we cannot assume it will use a prepended homolog. We therefore add a short continued-pretraining stage on pairs of the form (homolog, separator, target window), which teaches the model to copy across a relative while allowing for divergence. We report results with and without this stage. If JanusDNA weights are available, we repeat the main comparison with it to check that the effect does not depend on the architecture.
We then score variants under each aggregation scheme. For every variant, we log the per-vote scores, so we can estimate $\rho$ directly. We measure it as the correlation of per-vote errors against labels, across variants, separately for each source of votes.
Baselines
The naive reference point is HyenaDNA scoring each variant once, with no retrieval, as the log-likelihood difference between the alternate and reference windows. Every gain is measured from this point.
The strongest existing method is the Nucleotide Transformer family on ClinVar zero-shot pathogenicity. The Nucleotide Transformer reports 0.80 AUC. Benchmarking DNA foundation models reports 0.73 AUC as the best score, from NT-v2. The two figures almost certainly reflect different variant subsets and protocols, so we will rerun NT-v2 on our exact ClinVar split, with the same windows and the same reference-versus-alternate likelihood scoring. That rerun is the bar, not either published number. Beating it means a higher AUC, with a bootstrap confidence interval that excludes the NT-v2 value, using a model of comparable or smaller size.
The no-memory self-consistency baseline aggregates K perturbed views of the query alone: shifted windows, reverse complement, and dropout samples. It uses the same K and the same aggregation rule, which isolates whether memory matters, rather than whether voting does.
The joint-memory baseline concatenates all K homologs into one long context, in the style of E1 fusion and made feasible by HyenaDNA's long window. It isolates whether keeping the votes separate matters.
The kNN-LM baseline mixes HyenaDNA's base distribution at the variant position with neighbor distributions built from homolog hidden states. The mixing weight is tuned on a held-out split.
The count-only baseline aligns the retrieved homologs to the query with a pairwise aligner and scores a variant by how often the alternate base appears at that aligned position. It isolates whether the language model adds anything beyond the homologs themselves.
All variants share one retrieval index, the same K values from 1 to 64, and the same tuning budget of about twenty configurations each on a validation split.
Why not just…?
Why not just sample the model more times, as ordinary self-consistency does? Because its errors are expected to be correlated, the variance formula above predicts a plateau at small K. The prior evidence is BiasTrace and Scaling Test-Time Compute Without Verification or RL is Suboptimal. The no-memory baseline tests this directly. If its measured $\rho$ is as low as that of homolog-conditioned votes, and its AUC tracks ours, the memory component is unnecessary and we will say so.
Why not just put every homolog into one long context, since HyenaDNA can read a million bases? This is the real threat. E1's joint encoding works well for proteins, and joint attention could weigh the relatives against each other better than averaging can. Against it stands the finding in Lost in Tokenization that raw sequence can drown the signal, and the possibility that a single noisy homolog contaminates a joint pass but not a separate vote. The joint-memory baseline at matched K and compute decides between them. If joint fusion wins, the contribution narrows to the measurement of $\rho$, and the method recommendation becomes fusion.
Why not just count bases in the aligned homologs, which is ordinary conservation scoring? If the count-only baseline matches us, the language model is acting as an expensive aligner. We expect the model to help mainly where alignment fails or where context changes how much a base matters, for example near splice sites and in fast-evolving non-coding sequence. We will therefore report AUC broken down by homolog divergence and by region type, as well as overall.
Why not just use a k-mer or byte-pair model such as DNABERT-2? With those tokenizers, the variant changes which token appears, so each vote has no shared four-way coordinate on which to agree. LDARNet also reports that fixed single-base grids beat adaptive tokenizations at localizing compact splice signals. We still include NT-v2 as the tokenized competitor, scored by likelihood difference.
Proposed Experiments and Decisive Checks
The primary benchmark is ClinVar zero-shot pathogenic-versus-benign AUC. We add an HGMD-based split for transfer; no reported result found on Lacuna for HGMD scoring numbers. As a secondary benchmark we use the lenti-MPRA functional readout used by BMFM-DNA, a massively parallel reporter assay that measures how sequence variants change gene expression; no reported result found on Lacuna for lenti-MPRA.
The decisive check has three parts. First, AUC should rise with K for homolog voting beyond the point where the no-memory curve flattens. Second, the measured $\rho$ for homolog votes should be clearly lower than for perturbation votes. Third, the gap between the two curves should be predictable from the formula using the measured $\rho$ values. If all three hold, the mechanism is confirmed. If AUC improves but $\rho$ does not drop, the gain comes from somewhere other than decorrelation, and our account is wrong.
We also report selective-prediction curves, ranking variants by vote margin, to test whether agreement among homolog votes is a calibrated signal for abstaining on uncertain variants. Finally, an ablation with and without continued pretraining tells us whether in-context use of homologs must be trained in, or whether the kNN-LM route is the practical one.
References
- Solving Quantitative Reasoning Problems with Language Models
- Towards Reasoning in Large Language Models: A Survey
- BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs
- Scaling Test-Time Compute Without Verification or RL is Suboptimal
- E1: Retrieval-Augmented Protein Encoder Models
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
- Generalization through Memorization: Nearest Neighbor Language Models
- HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
- SegmentNT: annotating the genome at single-nucleotide resolution with DNA foundation models
- JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model
- BMFM-DNA: A SNP-aware DNA foundation model to capture variant effects
- Lost in Tokenization: Context as the Key to Unlocking Biomolecular Understanding in Scientific LLMs
- The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics
- DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes
- Benchmarking DNA foundation models for genomic and genetic tasks
- LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling