Cutting the Tree Out of the Protein Language Model

A protein language model — a Transformer trained on millions of natural sequences, in the same way text models are trained on language — learns what makes a protein viable. But its training corpus is not a random sample of protein space: it is a tangled family tree, heavily skewed toward a few model organisms, in which most sequences are near-duplicates of their close relatives. Recent evidence shows that as these models scale, they memorize exactly this tree: phylogenetic drift and sampling skew get baked into the weights alongside genuine fitness constraints, which is why ESM-2's raw likelihood scores plateau on variant effect prediction. The obvious fix, Likelihood-Fitness Bridging (LFB), averages scores over retrieved homologs at inference time to marginalize the drift away — but it needs alignments and homologs, so it fails precisely where conservation bias hurts most: in lineage-specific, orphan, and rapidly evolving sequences that have no deep homolog set to average over.

Meanwhile, an entirely separate literature on debiasing language and vision models has converged on a striking finding: learned biases — gender stereotypes, spurious shortcuts, dataset artifacts — concentrate in a tiny, surgically removable subset of weights. Zeroing as few as three neurons, or masking a sparse subnetwork, removes the bias while preserving the task, at a fraction of the cost of retraining. Nobody has tested whether this localization property holds when the "bias" is evolutionary: is there a small, prunable set of units inside a protein language model that encodes phylogeny rather than function? This proposal argues yes, and that finding it yields a debiasing method that is cheaper than retrieval-based approaches and, uniquely, still works when the homologs run out.

Core Claim

Phylogenetic drift memorization in protein language models is localized — concentrated in identifiable neurons and attention heads, separable from the fitness signal — and can therefore be removed by cheap internal weight surgery rather than external homolog retrieval. The testable consequence: a sparse, structured intervention inside a frozen ESM-2 should recover most or all of LFB's accuracy gains on ProteinGym while requiring a single forward pass with no database lookup, and it should retain those gains on proteins where LFB degenerates to the raw baseline because homolog retrieval returns nothing useful.

The claim has a deliberately falsifiable edge. The debiasing literature also proves a "bias starvation" limit: when a spurious feature is 99.9% correlated with the label, the network never learns the robust feature, and no subnetwork of it exists to extract. Deep evolutionary conservation is exactly such a near-total correlation at functional sites. So the proposal must also answer a sharper question: when we remove clade-predictive units, do we destroy fitness signal at ultra-conserved positions? Our working hypothesis is that we do not, because the surgery targets clade membership information (which lineage a sequence came from, and how heavily that lineage was sampled) rather than conservation itself — the two are confounded in the corpus but not identical in the representations — but this is an empirical question the program is built to settle, not assume.

Research Program

The first work package builds the localization instrument. Following the design of Finding Fantastic Weights, which attached an auxiliary "privacy head" to predict a bias attribute from frozen representations, we attach a phylogeny head: a small linear probe trained to predict the taxonomic lineage of a sequence from ESM-2's residue and sequence embeddings, using UniProt metadata as the label. Clade-predictive accuracy of the frozen latents is the measurement of drift memorization. We then rank internal units by their contribution to clade prediction using the attribution machinery of CRISPR (Integrated Gradients contrasting clade-label attribution against a fitness-relevant attribution) and of Debiasing Masks (frozen-weight movement pruning with a focal loss), yielding candidate "drift neurons" and drift-heavy attention heads.

The second work package performs and evaluates the surgery. We zero or down-weight the identified units under a structured sparsity budget — whole attention heads and feed-forward channels, not individual weights, because unstructured masks give no real inference speedup on standard hardware, a limitation the movement-pruning work itself acknowledges. The intervention objective is:

$$\min_{m}\; \mathrm{CladeAcc}\big(f_{m}(x)\big) \quad \text{s.t.} \quad \ell\big(f_{m}(x^{\mathrm{wt}}\!\to x^{\mathrm{mut}})\big) \approx \ell\big(f(x^{\mathrm{wt}}\!\to x^{\mathrm{mut}})\big)$$

where $m$ is the structured mask, $f$ the frozen model, and $\ell$ the mutant-versus-wild-type log-likelihood ratio used for variant scoring. In words: make the representation uninformative about which lineage a sequence belongs to, while barely perturbing the model's likelihood judgments on mutations. Calibration will use a deliberately tiny budget — on the order of tens of sequences, matching CRISPR's demonstration that 20 calibration samples suffice — because the whole point is a method a single lab can run.

The third work package confronts the starvation tension head-on. Using the multiple sequence alignments accompanying each ProteinGym assay, we stratify every mutated position by conservation level and every assay by alignment depth and taxonomic evenness. We ask where surgery helps (shallow, clade-skewed alignments; poorly conserved positions), where it is neutral, and where, if anywhere, it damages predictions at ultra-conserved catalytic positions — the regime in which the debiasing theory predicts failure because no separable robust feature exists.

Proposed Experiments

The primary benchmark is the ProteinGym substitution suite — 217 deep mutational scanning assays, roughly 2.4 million variants, scored by Spearman rank correlation between predicted and measured fitness. The reference numbers on this suite come from the LFB paper: raw zero-shot ESM-2 (15B) reaches a Spearman $\rho$ of 0.398, and adding LFB's homolog-averaging lifts it to 0.436; ProGen2 (6.4B) goes from 0.389 to 0.433. Our headline target is a masked ESM-2 that matches or exceeds 0.436 with one forward pass and no MMseqs retrieval, at matched total development compute. We will run ESM-2 at 650M, 3B, and (where feasible) 15B to test whether drift memorization — and hence the benefit of surgery — grows with scale, as the LFB authors' scaling-curse account predicts.

The second benchmark is clinical variant classification: ProteinGym's ClinVar-based benchmark of pathogenic versus benign mutations, where raw ESM-2 (15B) reports an ROC-AUC of 0.895 and LFB raises it to 0.938. For the supervised regime we compare against CPT-1, which reports an AUROC of 0.934 on roughly 24,000 ClinVar variants across 1,298 genes, versus 0.874 for zero-shot ESM-1v — and, more tellingly for us, a specificity of 68% at 95% sensitivity where ESM-1v manages only 27%. The high-sensitivity operating point is where conservation-vs-clinical-truth confounds bite hardest, so we report specificity at 95% sensitivity throughout.

The third and most decisive experiment is the orphan stratum. We simulate lineage-specific and poorly sampled proteins by, for each assay, progressively deleting homologs from the retrieval pool (restricting MMseqs to ever-shallower or clade-restricted hits) and tracing how LFB's advantage decays toward the raw-likelihood baseline. The prediction that distinguishes this proposal from all prior work: the surgically debiased model's accuracy is flat across this axis, because it carries its debiasing in its weights rather than in a retrieved alignment. If surgery instead decays in lockstep with LFB, the localization claim is wrong and the proposal should die.

Compute accounting is a first-class measurement, not an afterthought: we report wall-clock and FLOPs per variant for raw likelihood, LFB (retrieval plus homolog-averaging passes), and the masked model, with structured head-level pruning so sparsity converts to real speedup.

Baselines

The do-nothing reference point is raw zero-shot likelihood scoring of the frozen model, which on ProteinGym's 217 DMS assays gives Spearman $\rho = 0.398$ for ESM-2 (15B) and 0.389 for ProGen2 (6.4B), as reported by the LFB paper. This baseline matters because it quantifies how much of the apparent skill of large protein models is just memorized tree, and any debiasing method must clear it by a margin that survives error bars across assays.

The strongest existing competitor is LFB itself — post-hoc, inference-time marginalization of phylogenetic noise by averaging log-likelihood ratios over retrieved homologs, achieving $\rho = 0.436$ on ProteinGym DMS and ROC-AUC 0.938 on the clinical benchmark with ESM-2 (15B). It is the strongest because it is the only published method that targets exactly our bias (phylogenetic drift in protein models) at exactly our price point (no weight updates). The comparison is run on identical checkpoints, identical ProteinGym splits and scoring conventions, and matched total compute including LFB's retrieval cost; our method wins only if it matches or exceeds 0.436 and 0.938 while being strictly cheaper per variant and — the decisive margin — retaining that performance on the orphan stratum where LFB's homolog pool is depleted. Anything short of the stratum result is a tie at best, since LFB already exists.

Two ablations isolate the key ingredient, which is phylogeny-targeted localization rather than pruning per se. First, matched-sparsity random pruning: remove the same fraction of attention heads and channels chosen at random; if this does as well, the story is just regularization, not drift localization. Second, scrambled-label surgery: run the full pipeline with randomly permuted clade labels; if this also lifts DMS performance, the phylogeny head is measuring nothing specific. Both run with the same calibration-sample budget (tens of sequences, as CRISPR used 20) on the same checkpoints. We also report the supervised reference CPT-1 (AUROC 0.934 on ClinVar) as context for what a completely different, DMS-supervised approach achieves, though it occupies a different resource regime since it requires labeled mutational scanning data from five proteins.

Why not just…?

Why not just use LFB — homolog averaging already works and costs no weight updates? This is the most serious objection, and we treat it as the co-primary hypothesis rather than a foil. The file's own accounting of LFB's limits is our evidence: the method "cannot occur" for taxonomically restricted, orphan, or de novo sequences lacking retrieved homologs, and its Ornstein-Uhlenbeck drift assumption breaks under lineage-specific adaptation. The orphan-stratum experiment is designed precisely to measure where LFB stops working; if surgical debiasing does not hold its advantage there, we will report that LFB should remain the method of choice and this line closes. We are honest that on proteins with deep, even alignments, LFB may simply be better, and we will say so.

Why not just retrain or fine-tune the whole model on balanced, lineage-aware data? The computational-perspective survey of fairness in deep learning documents that full-parameter optimization of modern foundation models is computationally prohibitive and impossible for closed-weight systems, and the adapter work shows full fine-tuning suffers "fairness forgetting" — later task tuning reinstates the original bias — whereas a 885K-parameter debiasing adapter on a frozen 110M model cut bias from 1.33 to 0.39 while retaining downstream accuracy. The biological version of this problem is worse: lineage-specific functional elements are by definition scarce, and CPT-1's authors show experimental mutational data exists for only a tiny slice of the proteome, which is why they train on five proteins rather than retrain anything. Balanced-data retraining fails on both axes this proposal cares about: cost and data availability.

Why not just drop the conservation filter and score with raw sequence likelihood? The literature directly rules this out at scale. Removing conservation filters in miRNA target prediction floods the genome with false positives, and raw pLM likelihoods suffer the scaling curse: ESM-2 (15B) manages only $\rho = 0.398$ on ProteinGym precisely because the likelihood memorizes drift, while ESM-1v's zero-shot clinical specificity at 95% sensitivity collapses to 27%. The problem is not the filter — it is that the corpus and the tree are inseparable by naive scoring, which is why the debiasing has to be either external (LFB) or internal (ours).

Why not just use classical phylogenetic models on trees, which already model lineage-specific divergence? Tree-based methods are principled but computationally and structurally cornered: tree reconstruction is NP-hard on modern alignment sizes (FastTree is prohibitive at ~450,000 sequences, while the cherry-pairing approximation needed 10–100× speedups to cope), the methods require reliable multiple sequence alignments that intrinsically disordered regions — roughly 40% of the proteome — cannot provide, and they cannot handle insertions and deletions. FastCherries buys speed by assuming site independence, giving up the epistasis a large model captures. Our approach keeps the model's epistatic capacity and removes only its tree, which is the reverse trade.

Decisive Checks

Three findings would settle the proposal. First, the phylogeny head should predict clade from frozen ESM-2 embeddings well above chance, and the identified drift units should be a small, structured set — if clade information is diffusely distributed across all layers, surgical debiasing is dead on arrival and we report that as a negative result with the localization maps. Second, on the ProteinGym strata, surgery should help most where alignments are shallow and clade-skewed and least where they are deep and even, confirming the mechanism rather than a generic regularization effect. Third, at ultra-conserved positions — the bias-starvation regime where conservation and function are nearly perfectly correlated — the masked model's mutation scores should not degrade; if they do, the separability hypothesis fails and the correct conclusion is that drift and fitness are entangled at exactly the sites clinical genetics cares about most, which would itself reshape how the debiasing-cost literature's starvation limit should be read in a biological setting. Each check is a standalone contribution, and none of them can come out uninformative.