Does Low-Rank Fine-Tuning Carry Exome Bias into DNA Models?

Most labeled human genetic variants sit in the 1.5–2% of the genome that codes for protein. The rest of the genome, which contains promoters, enhancers and deep introns, has few curated labels, even though it holds many real disease causes that exome sequencing misses. When a DNA language model (a model pre-trained to predict masked nucleotides, then adapted to tasks) is fine-tuned to call variants harmful or harmless on these labels, it can learn a cheap rule: "coding means pathogenic, non-coding means benign." That rule scores well on standard test sets and fails on the non-coding variants where the model is most needed. The usual adaptation method is low-rank adaptation (LoRA), which freezes the model and learns a small update to each weight matrix as the product of two thin matrices. We argue that LoRA makes this shortcut more likely. A rank-limited update tends to spend its few directions on whatever feature best separates the labels, and on exome-heavy data that feature is coding-versus-non-coding identity.

The proposal has two parts. First, a measurement: for LoRA adapters trained on variant labels, how much of the prediction comes from region identity, and does that share grow as the rank shrinks? Second, a fix, if the measurement confirms the problem. We fit a linear probe for "is this position coding?" on the frozen model, then constrain every LoRA update to write only outside the probe's directions. The adapter can still learn variant-disruption features that coding and non-coding variants share. It cannot learn them through the region axis. The test is whether this improves accuracy within non-coding variants, on held-out causal expression variants, compared with the simpler fix of rebalancing the training data.

Core Claim

Our hypothesis is that exome-biased labels combined with a rank-limited update produce an adapter whose useful signal is concentrated on the coding/non-coding axis. If so, removing that axis from the update should raise non-coding performance more than reweighting data or restricting training to non-coding labels. The reason is that the projection keeps the abundant coding labels as a training signal while blocking the one feature that transfers badly. Our constrained update at layer $\ell$ is

$$\Delta W_\ell = (I - U_\ell U_\ell^\top)\, B_\ell A_\ell,$$

where $B_\ell A_\ell$ is the ordinary LoRA product and $U_\ell$ is an orthonormal basis for the top-$k$ directions of a coding-identity probe fit on layer $\ell$'s frozen hidden states. The factor $(I - U_\ell U_\ell^\top)$ removes any component of the update's output that lies along those directions. This is a claim about what low-rank updates learn from biased labels. It is not a claim that LoRA helps genomics in general, which has already been shown.

Prior Work and Positioning

The motivation comes from the exome-bias literature. ExAC (Analysis of protein-coding genetic variation in 60,706 humans) catalogued coding variation in 60,706 people and deliberately left non-coding sequence out of scope. CCDS (The consensus coding sequence project; The consensus coding sequence (CCDS) project: Identifying a common protein-coding gene set for the human and mouse genomes) defined coding boundaries while excluding UTRs, and we reuse those boundaries as our region labels. DeepVRegulome (DeepVRegulome) found that most high-impact recurrent non-coding glioblastoma mutations are absent from ClinVar. That is direct evidence that the labels a fine-tuned model sees are skewed toward coding sequence.

Several papers use LoRA or adapters in genomics but none ask what the adapter learns. DNABERT-2 (DNABERT-2) ships LoRA as a memory saver and evaluates on local-window classification. Genome-Factory (Genome-Factory) shows that LoRA and adapters match full fine-tuning on the GUE benchmark (28 local-window genomic classification datasets) and on related suites. Neither paper stratifies results by region or examines adapter directions, which is exactly what we do.

Two evaluation papers constrain the design. The gene-centric causal-variant study (Gene-centric evaluation of causal variant prediction for DNA models) and the 57-task benchmark (Benchmarking DNA foundation models for genomic and genetic tasks) evaluate only frozen embeddings. The 57-task benchmark finds a "specialization gap": frozen models do reasonably on pathogenic-versus-common variants but poorly on expression variants. We read that pattern as consistent with a constraint-and-region shortcut, and we test it after fine-tuning, which those papers explicitly did not do. The Mechanistic Invariance Test (The Mechanistic Invariance Test) shows that these models lean on AT content rather than positional grammar. Our proposal is a label-level analogue of that finding, with coding identity as the confound instead of nucleotide composition.

Two papers supply the method and one suggests a counter-hypothesis. The LoRA-variants study (A Unified Study of LoRA Variants) shows that vanilla LoRA with a proper learning-rate sweep matches elaborate variants, so we build on vanilla LoRA and sweep learning rates. Face-forgery work (Enhancing General Face Forgery Detection via ViT with LoRA) reports that LoRA resists dataset-specific artifacts. That runs against our hypothesis, and we treat it as a live possibility. If LoRA at low rank shows less region reliance than full fine-tuning, our shortcut story is wrong and the paper reports that.

Research Program

Backbones. We use three pre-trained DNA models: DNABERT-2 (117M parameters, with LoRA support built in), Caduceus, and Nucleotide Transformer v2. Training runs through Genome-Factory so that full fine-tuning, LoRA and adapters share one code path.

Training labels. We train on the pathogenic-versus-common variant task used in the 57-task benchmark. Each variant is labeled coding or non-coding using CCDS boundaries, and we report the region mix of each class, which we expect to be heavily coding among pathogenic variants.

Stage 1: the audit. We train LoRA at ranks 2, 4, 8, 16 and 64, plus full fine-tuning, sweeping learning rates for each. For every model we compute three quantities:

  • pooled AUC on held-out variants;
  • AUC separately within coding variants and within non-coding variants;
  • a region-reliance index, defined as the drop in pooled AUC when predictions are replaced by region-conditional residuals. Concretely, we regress the model's score on region identity and evaluate only what is left over.

We also measure how much of each adapter's output variance lies in the probe directions $U_\ell$ and in randomly chosen directions. The prediction is that region reliance increases as rank decreases.

Stage 2: the constrained adapter. At each adapted layer, we fit the coding-identity probe on frozen hidden states computed from windows around CCDS-annotated and non-coding positions that are not in any evaluation set. We take the top-$k$ probe directions, for $k$ in {1, 4, 16}, and apply the projection above to both the LoRA updates and the classification head.

Projection carries a real risk. Some true biology is region-dependent, for example splice-disrupting variants near exon boundaries. We therefore report splice-proximal variants as a separate stratum, to check that the projection does not delete genuine signal.

Secondary measurement. Motivated by On the Calibration of Large Language Models and Alignment, we measure expected calibration error separately within each region. We are testing whether LoRA's reported preservation of calibration also holds for non-coding variants, where labels are sparse.

Baselines

Naive reference points. The simplest reference is a score that uses only region identity. Its pooled AUC tells us how much of the benchmark is solvable by the shortcut alone. The second reference is the frozen backbone with mean-pooled embeddings and a shallow classifier, the protocol of Benchmarking DNA foundation models. That paper reports a pathogenic-variant AUC of 0.73 for Nucleotide Transformer v2. On the non-coding expression-variant task of Gene-centric evaluation, the same kind of frozen protocol gives variant-centric AUC 0.722 for Nucleotide Transformer and 0.706 for HyenaDNA, and a 1.5 kb convolutional network reaches 0.695.

Strongest existing method. The strongest existing method is the supervised long-context model Enformer, which reaches gene-centric AUC 0.764 in Gene-centric evaluation, against 0.703 for Caduceus and 0.670 for HyenaDNA. No reported result found on Lacuna for within-non-coding AUC on the pathogenic-versus-common task for any model, and none for any LoRA-tuned model on the GTEx causal-variant benchmark. We will produce those numbers ourselves.

Fair comparison. Every learned baseline shares the same variant splits, a chromosome-held-out split, and an equal number of learning-rate sweep points. LoRA methods use equal trainable-parameter budgets. Full fine-tuning and data-rebalanced LoRA use the same training labels as our method.

Ablations. Three ablations isolate the key ingredient. The first projects out a random $k$-dimensional subspace. The second projects out the region directions only at the classification head. The third tests LoRA trained on non-coding labels only.

Win condition. "Beating" the baselines means a higher within-non-coding AUC on the chromosome-held-out split, and on GTEx causal variants at the variant-centric level, than both rebalanced LoRA and unconstrained LoRA. It also requires that the gain does not come from lower within-coding AUC alone. As a stretch goal, gene-centric AUC should approach Enformer's 0.764.

Why not just…?

Why not just rebalance the training data by region? This is the most serious threat. Upweighting non-coding examples, or sampling equal numbers from each region, directly weakens the coding-means-pathogenic correlation. Our argument is that the bottleneck is scarcity, not proportion. Upweighting a few hundred non-coding positives tends to overfit them. The projection instead keeps all coding examples as a learning signal and only forbids the region axis. The within-non-coding comparison against region-balanced LoRA, at matched tuning budget, settles the question. If rebalancing matches our method, we will report the audit as the main result and recommend rebalancing.

Why not just train only on non-coding labels? This removes the shortcut by construction but throws away most of the labels. The non-coding-only ablation measures how much the coding data contributes once the region direction is blocked. If this ablation wins, our transfer argument fails.

Why not just fully fine-tune? Genome-Factory and the LoRA-variants study show full fine-tuning costs about three times the throughput and far more memory, but cost is not our argument. Full fine-tuning is included in the rank sweep. If it shows lower region reliance than low-rank LoRA, that supports our mechanism and makes it a strong competitor on accuracy. If it shows higher region reliance, the face-forgery result generalizes to this setting and our hypothesis is refuted.

Why not just use frozen embeddings, or Enformer? Frozen embeddings trail Enformer on expression variants in both evaluation papers, and neither paper tested whether fine-tuning closes that gap. Enformer requires heavy supervised training on expression data. Our method needs only variant labels plus a frozen backbone. Both approaches appear as reference numbers rather than being argued away.

Decisive Experiments

The first decisive check is the rank sweep of the region-reliance index. If region reliance does not rise at low rank, and if LoRA adapters do not concentrate their output variance in the probe directions more than in random directions, the shortcut hypothesis is false and Stage 2 is unmotivated.

The second check is within-non-coding AUC on the chromosome-held-out split and on the GTEx causal variants (fine-mapped with SuSiE, posterior inclusion probability above 0.9), at both the variant-centric and gene-centric levels. We compare projection LoRA, random-projection LoRA, rebalanced LoRA, non-coding-only LoRA and full fine-tuning. The projection must beat the random-subspace control, or the gain is just generic regularization.

The third check is external transfer to a non-coding task with no region shortcut available. We use DeepVRegulome-style splice and transcription-factor binding disruption scores, measured by in-silico mutagenesis (scoring each variant as the change in prediction between reference and mutated sequence). Splice-proximal variants are reported separately, so that any damage from removing region information is visible rather than averaged away.