DNA models trained across many species are good at one thing and poor at another. They recognize which mutations evolution would not tolerate, and so they flag disease-causing variants well. They are weak at predicting how much a variant changes gene activity in a particular tissue. Specialized models trained directly on laboratory measurements of chromatin and expression have the opposite profile. We propose to train one sequence model on both signals at once: an evolutionary objective computed over hundreds of mammalian genomes, and a regression objective on human and mouse functional-genomics tracks. The new ingredient is a per-position rule that detects where the two objectives push a nucleotide's representation in opposite directions and removes the evolutionary push at that position only.
The motivating hypothesis is concrete. Many active human regulatory elements are young or only weakly conserved. A conservation objective therefore teaches the model to treat them as neutral DNA, while assay data says they matter. If this localized disagreement is what makes cross-species models fail on quantitative regulatory variants, then resolving it position by position should give a single model that keeps the evolutionary model's variant-pathogenicity scoring and gains the assay model's regulatory accuracy. It should also carry regulatory mechanism to a species whose tracks were never used in training.
Core Claim
The claim has three parts. First, during joint training, the gradient disagreement between an evolutionary-constraint loss and an assay-regression loss is not spread evenly across the genome. It concentrates in lineage-specific, active regulatory sequence. Second, a cheap activation-level correction removes that disagreement where it occurs and leaves agreement intact elsewhere. The correction projects out the component of the evolutionary gradient that opposes the assay gradient, computed on each position's hidden vector rather than on the full parameter vector. Third, the resulting model matches or beats the better of a late-fusion ensemble of the two specialist models on both families of benchmarks. If the ensemble does as well, the joint-training story is wrong, and we will say so.
Grounding in Prior Work
The gap comes from benchmark evidence. Benchmarking DNA foundation models for genomic and genetic tasks and DART-Eval found that multi-species language models such as NT-v2, Caduceus and DNABERT-2 do well on pathogenic-versus-benign variant classification but fail on quantitative regulatory variants, meaning variants that shift expression or chromatin accessibility. DART-Eval also reports that a 6.6M-parameter supervised convolutional model, ChromBPNet, is not beaten by these large pre-trained models on regulatory tasks.
A Phylogenetic Approach to Genomic Language Modeling sharpens the gap and supplies one of our two objectives. PhyloGPN trains a reverse-complement-equivariant network on a 447-mammal alignment to predict per-site substitution-model parameters, and it reaches 0.96 AUROC on BEND disease variant effect prediction. The authors report that the same model fails on expression-variant effects. We read that failure as a fitness-versus-mechanism mismatch and treat it as testable.
Enformer (Effective gene expression prediction from sequence by integrating long-range interactions) supplies the other objective. It jointly regresses 5,313 human and 1,643 mouse tracks over 200 kb inputs and shows that sharing regulatory logic across the two species helps. Enformer, however, uses no evolutionary signal beyond those two species, cannot address species without tracks, and sums task losses without any conflict handling. Our proposal differs by adding a 447-species constraint signal and by making the interaction between the two objectives the object of study.
The nearest unified multi-task system is Omni-DNA. It casts many genomic classification tasks as instruction-following next-token prediction after multi-species pre-training, and it reports multi-task gains (0.767 versus NT-2.5B's 0.709 on the Nucleotide Transformer benchmark). Omni-DNA is unidirectional, it does not train the evolutionary objective jointly with assay regression, and it has no conflict mechanism. annDNA injects annotation knowledge into a human-only model through distillation. It is complementary to our work but contains no phylogenetic signal.
For method, RotoGrad supplies the diagnosis: negative transfer comes from both mismatched gradient magnitudes and opposing gradient directions. It fixes these with learned task-specific rotations of shared features, tested only on vision. RotoGrad's corrections are global per task and expensive at high dimension. Our hypothesis is that the conflict here is local in the genome, which motivates a per-position correction instead.
For the backbone we reuse Caduceus (Caduceus). It is a bidirectional, reverse-complement-equivariant state-space model whose cost grows linearly with sequence length, and it already beats Enformer on long-range variant effect prediction. It was pre-trained only on the human reference, and its authors name multi-species training as future work. That is the gap this proposal fills.
Research Program
We build one Caduceus backbone with a context on the order of 100 kb and two heads. The phylogenetic head follows PhyloGPN. At every aligned human position, it predicts substitution-model parameters, scored by likelihood of the Zoonomia column under the mammalian tree. The assay head predicts binned coverage for a manageable subset of Enformer's human and mouse tracks, namely DNase/ATAC and CAGE. Two specific mechanisms are used. First, for each position $i$ we compute the gradients of the two losses with respect to that position's hidden vector $h_i$. Second, wherever their cosine is negative, we remove the component of the phylogenetic gradient that lies along the assay gradient before backpropagating into shared weights:
$$ g^{\text{phy}}_i \leftarrow g^{\text{phy}}_i - \min\!\left(0, \frac{\langle g^{\text{phy}}_i, g^{\text{assay}}_i\rangle}{\lVert g^{\text{assay}}_i\rVert^2}\right) g^{\text{assay}}_i $$
Here $g^{\text{phy}}_i$ and $g^{\text{assay}}_i$ are the two loss gradients at $h_i$. This adds roughly one extra backward pass through the heads, and it scales with sequence length rather than with parameter count. That is how we sidestep the high-dimensional rotation cost that RotoGrad faces. Magnitude balancing uses a single scalar reweighting per task.
The analysis half matters as much as the model. We log the fraction of positions with conflict, and the conflict strength, stratified by element class. The classes are ENCODE-style candidate regulatory elements split by conservation, coding exons, splice sites, and background. This tests the first part of the claim directly. If conflict is not enriched in weakly conserved active elements, the motivating mechanism is false, and we report the method as a generic optimizer at best.
Baselines
The naive reference point is the same backbone trained on the summed losses with a fixed weight tuned on validation chromosomes. If our correction does not beat this, the joint-training result reduces to "more data helps."
The strongest regulatory competitor is a supervised specialist trained on the same assay tracks alone. We use a Caduceus backbone of matched size trained only on the assay head, and we also report ChromBPNet as described in DART-Eval. For DART-Eval's quantitative and variant tasks, the file's excerpt contains the claim that ChromBPNet is not beaten but no specific figure, so no reported result found on Lacuna for ChromBPNet's DART-Eval caQTL score. We rerun it ourselves on identical splits. For expression we cite Enformer's reported 0.85 Pearson on TSS CAGE (Enformer), but we compare primarily against our matched-compute assay-only model, since we use fewer tracks.
The strongest constraint competitor is PhyloGPN. Its reported 0.96 AUROC on BEND disease variant effect prediction is the bar our joint model must stay within noise of. Nucleotide Transformer's ClinVar AUC of 0.80 (The Nucleotide Transformer) is a secondary reference.
The sequential baseline pre-trains on the phylogenetic loss and then fine-tunes on assay tracks. The fusion baseline trains the two specialists separately and combines their variant scores with a logistic layer fit on validation data. RotoGrad applied to the pooled shared features is the global conflict-resolution baseline.
Two ablations isolate the key ingredient. The first applies the projection at a random set of positions matched in fraction to the detected conflicts. The second gates the phylogenetic loss by a fixed conservation threshold instead of by measured disagreement.
All models get the same data, the same backbone size, the same token budget, and the same number of hyperparameter trials (about twenty per method), selected on held-out chromosomes. Beating a baseline means a gain larger than the spread across three seeds, on held-out chromosomes, without losing more than that spread on the other benchmark family.
Why not just…?
Why not just fine-tune a cross-species pre-trained model on the assay tracks? This is the standard recipe. The evidence in hand suggests it is insufficient: NT-v2 and Caduceus fine-tuned or probed on regulatory tasks trail ChromBPNet in DART-Eval. Sequential training can also overwrite what pre-training learned, the forgetting concern raised by Wang et al.. Our sequential baseline measures this directly. If it keeps PhyloGPN-level disease AUROC while matching the assay-only model, the joint method is unnecessary.
Why not just ensemble PhyloGPN with a supervised regulatory model? This is the real threat, because two specialists that each win their own benchmark may simply add up. Our method must beat fusion somewhere fusion cannot reach. The candidate is the held-out-species test below, where the assay specialist has never seen the species' tracks. A second candidate is regulatory variants inside weakly conserved elements, where neither specialist alone is informed. If fusion ties us on both, we will conclude that joint training adds nothing beyond model combination.
Why not just use RotoGrad or a fixed loss weight? Fixed weights address scale but not direction. RotoGrad addresses direction, but with one rotation per task over the whole feature space, which cannot express "agree in exons, disagree in young enhancers." If our stratified conflict analysis shows the disagreement is uniform across element classes, RotoGrad should do as well, and the head-to-head comparison will show it.
Proposed Experiments and Decisive Checks
All models train with chromosome-held-out splits. Evaluation covers the five DART-Eval regulatory tasks, with emphasis on quantitative activity and counterfactual variant effect. Variant scoring always compares single-nucleotide substitutions at base resolution, avoiding the tokenization artifact DART-Eval documents. We also evaluate BEND disease variant effect prediction, held-out-chromosome Pearson correlation on human tracks, and frozen-embedding probing on GENEB, including its 1-shot and 10-shot regimes, where it reports most models collapse.
The decisive cross-species check trains the assay head on human tracks only and evaluates on held-out mouse tracks. Mouse is present in the 447-mammal alignment but never supplies assay labels. Joint training should transfer regulatory activity to mouse better than both the human-only assay model and the fusion ensemble.
The decisive mechanistic check stratifies DART-Eval variant performance by element conservation. Our predicted signature is that the gain over the naive summed-loss model concentrates in weakly conserved active elements, which is where the logged conflict concentrates. A gain that is uniform, or that appears where no conflict was logged, would falsify the mechanism even if aggregate numbers improve.
References
- Benchmarking DNA foundation models for genomic and genetic tasks
- DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA
- A Phylogenetic Approach to Genomic Language Modeling
- Effective gene expression prediction from sequence by integrating long-range interactions
- Omni-DNA: A Unified Genomic Foundation Model for Cross-Modal and Multi-Task Learning
- annDNA: Learning Annotation-Aware Genomic Representations via Knowledge Distillation
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
- The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics
- Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
- GENEB: Why Genomic Models Are Hard to Compare