Clade Bias in Protein Embeddings: One Projection, Audited on the Graph

Protein language models — Transformers trained on hundreds of millions of raw amino-acid sequences — turn every protein into a vector, and searching that vector space now beats classical sequence alignment at finding distant relatives. But the sequence databases these models learn from are spectacularly uneven: a few organisms and a few gene families dominate, while most of the tree of life is barely sampled. In text models, uneven training data leaves a measurable geometric scar: stereotype directions that can be found with principal component analysis and removed with a single linear projection, as shown for gender bias in word embeddings by Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings and refined into a one-step method by Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection. Nobody has checked whether the same is true for proteins.

We propose to treat taxonomic overrepresentation — "clade bias" — exactly the way the fairness literature treats demographic bias, and to audit the result where bias should be most visible: the topology of the nearest-neighbor graph over embeddings. If over-represented clades warp the geometry, they should create "hub" proteins that become the nearest neighbor of far too many queries; graph degeneracy methods, which were built precisely to separate cohesive cores from peripheral high-degree hubs in word networks (A Graph Degeneracy-based Approach to Keyword Extraction), give us a principled audit. The payoff would be a lightweight correction — one projection, applied to a frozen model — that improves retrieval for under-sampled proteins at zero retraining cost.

Core Claim

The claim has two halves. First, clade bias in frozen protein language model (PLM) embeddings is substantially linear: a small number of directions in the embedding space carry which lineage a sequence comes from, and a single Mean Projection onto the hyperplane orthogonal to the difference of clade means removes most of it. Second, the bias is topologically legible: in the $k$-nearest-neighbor graph over embeddings, spurious hubness concentrates in over-represented clades, degree and PageRank reward that hubness, and $k$-core decomposition — recursively peeling away nodes whose connections are too sparse to form a dense cluster — separates genuine functional cores from study-bias artifacts. Removing the linear component should flatten the spurious hubs and improve retrieval for under-represented families, without retraining anything.

The projection itself is borrowed unchanged from the text setting. Let $\mu_A$ and $\mu_B$ be the mean embeddings of an over-represented clade $A$ and the remainder $B$, let $\mu$ be the global mean, and set the bias direction $w = \mu_A - \mu_B$. Each embedding $x$ is replaced by

$$\hat{x} = x - \frac{(x - \mu)\cdot w}{\lVert w \rVert^2}\, w,$$

which zeroes the component along $w$ while otherwise preserving the vector. Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection shows this single step matches or beats ten-to-fifteen rounds of iterative nullspace projection (INLP) on text, cutting biography-classification gender bias (TPR-GAP) by 99.4% versus 35% for INLP, and altering only 8 of 120 nearest neighbors versus 43 for INLP — that neighbor-preservation property is exactly what we need, since our downstream use is nearest-neighbor retrieval.

Research Program

Three components, in dependency order.

Audit. We reproduce the retrieval recipe of Nearest neighbor search on embeddings rapidly identifies distant protein relations: mean-pooled ProtT5 and ESM-1b embeddings, an approximate-nearest-neighbor index, evaluated on CATH20 (14,433 single-domain sequences below 20% pairwise identity) and Pfam20 (full-length, multi-domain proteins). On the resulting $k$-NN graph we measure hubness (how often each protein appears in others' neighbor lists), degree, PageRank, and $k$-core membership, and annotate every node with its clade's representation in PLM training databases. The diagnostic question: do degree and PageRank concentrate on over-represented lineages, and does the innermost $k$-core recover CATH superfamily or Pfam family labels better than a degree ranking does? The analogy is direct: the degeneracy work on word networks showed PageRank is systematically misled by high-degree peripheral hubs, with $k$-core selection beating TextRank by more than 5.5% absolute F1 on short and medium documents (A Graph Degeneracy-based Approach to Keyword Extraction), and in protein interaction networks, degree is a known confounder that models memorize rather than real biology (Fast Proteome-Scale Protein Interaction Retrieval via Residue-Level Factorization; the same lesson dates to Lethality and centrality in protein networks, where centrality–essentiality claims had to survive ascertainment bias in which proteins get studied at all).

Correct. We compute clade-mean difference vectors at several taxonomic levels (phylum-level contrasts, and within-family dominant-versus-rest contrasts), apply one projection per contrast, and verify three things: a linear probe for clade identity drops toward chance (the guarding criterion); nearest-neighbor overlap with the unprojected space stays high (the preservation criterion, measured the way the Mean Projection paper counts altered neighbors); and hubness in the rebuilt $k$-NN graph drops preferentially in over-represented clades. A real risk, flagged below, is that clade identity is not linear in PLM space — the norm-frequency analysis of On the Curious Case of l2 norm of Sense Embeddings found that static sense embeddings satisfy $\lVert s \rVert_2^2 \propto \log\text{-frequency}$ but that contextualized models like BERT do not, so we cannot assume the text-side geometry carries over. We therefore also run a cheap norm audit: does pooled PLM vector length correlate with clade frequency, and does plain length normalization already remove part of the hubness?

Evaluate. Retrieval on CATH20 and Pfam20, stratified by clade representation; enzyme-function placement on the Unseen-EC4 benchmark of How Not to be Seen: Predicting Unseen Enzyme Functions using Contrastive Learning, where the interesting stratum is families with almost no training members; and a probe on the Observed Antibody Space paired subset, where Addressing the antibody germline bias and its effect on language models for improved antibody design showed antibody PLMs trained on germline-dominated repertoires hallucinate germline residues 85–95% of the time. We test whether a germline-versus-mutated mean direction exists in pooled embeddings and whether removing it shifts retrieval toward somatically mutated sequences — a probe of the geometry, not a claim to fix token-level generation.

Proposed Experiments

Experiment 1 (audit) is descriptive but decisive: if hubness does not concentrate in over-represented clades, the premise fails and we report that. Experiment 2 (projection plus retrieval) is the headline. On CATH20 the numbers to beat are taken from Nearest neighbor search on embeddings rapidly identifies distant protein relations: MMseqs2 sequence alignment alone reaches 34.5% ± 1.2% normalized top-1 accuracy, pooled-embedding retrieval alone 57.5% ± 1.6%, and the hybrid 62.0% ± 1.4%. On Pfam20 the ordering reverses for pooled embeddings — MMseqs2 achieves AUC1 of 0.520 versus 0.367 for pooled ProtT5 retrieval and 0.557 for the hybrid — which is itself part of our story: mean pooling degrades on multi-domain proteins, and we test whether part of that degradation is hubness from large over-represented families rather than pooling alone. Success is defined per stratum: matched or better overall accuracy, with a clear gain for under-represented clades. Experiment 3 runs nearest-neighbor enzyme placement on Unseen-EC4, where EnzPlacer reports 43.5% EC2 placement accuracy (38.5% for the CLEAN baseline, 30.3% for EnzPlacer below 10% sequence identity, 35.6% at EC3); our comparison is against a frozen-embedding nearest-centroid baseline of the CLEAN type, asking whether projection helps most in the sparse-family stratum. Experiment 4 is the antibody probe. Throughout, we track neighborhood preservation, because the danger mirror-image of Measuring Gender Bias in Word Embeddings of Gendered Languages Requires Disentangling Grammatical Gender Signals applies here too: in gendered languages, grammatical gender dominates vector geometry and masks the stereotype one is trying to measure; in proteins, clade identity is partly real biology (an organism's proteins genuinely co-evolve), so over-removal would destroy functional signal, and the neighbor-overlap metric is our guard against it.

Baselines

The do-nothing reference is frozen, mean-pooled ProtT5/ESM-1b embeddings searched exactly as in Nearest neighbor search on embeddings rapidly identifies distant protein relations, which reports 57.5% ± 1.6% normalized top-1 accuracy on CATH20 — this is the system our projection modifies, so any change is attributable to the projection alone. The strongest existing competitor is the MMseqs2-plus-embedding hybrid from the same paper (62.0% ± 1.4% on CATH20; AUC1 0.557 on Pfam20 versus 0.520 for MMseqs2 alone); we run it on identical data with no extra tuning, apply our projection only to the embedding leg, and count success as beating the hybrid overall or matching it while significantly improving the under-represented-clade stratum. For the diagnostic claims, the competing centrality baseline is degree and PageRank, run on the same $k$-NN graphs with the same annotation; the degeneracy literature's finding that PageRank rewards peripheral hubs while $k$-core selection wins by over 5.5% absolute F1 on text (A Graph Degeneracy-based Approach to Keyword Extraction) predicts degree will rank over-represented clades highly while cores track function, and our audit tests exactly that on proteins. Finally, two ablations isolate the key ingredient: a random-direction projection (project onto the hyperplane orthogonal to a random vector, matching the dimensionality change) — the Mean Projection paper showed apparent gains from multi-step debiasing can be mere dimensionality-reduction artifacts, and this ablation rules that out here — and length normalization, which isolates whether the cheap frequency-confound fix from the norm analysis explains the gains. For fitness-style scoring, the recalibration method of Removing bias in sequence models of protein fitness is a baseline in spirit; no reported result found on Lacuna for an aggregate scalar metric for it on the 14 deep mutational scanning datasets, so we treat it as qualitative, not numerical, there.

Why not just…?

Why not just use sequence alignment (MMseqs2, BLASTp)? Below 20% sequence identity on CATH20, MMseqs2 manages only 34.5% ± 1.2% versus 57.5% ± 1.6% for embedding retrieval, and below 10% identity alignment collapses while EnzPlacer still places 30.3% at EC2 — alignment cannot see the distant relations where clade bias matters most. The honest counterpoint is Pfam20, where MMseqs2 (AUC1 0.520) beats pooled embeddings (0.367): if our projection cannot narrow that gap on multi-domain proteins, the correction is only useful in the single-domain regime, and we will say so. The hybrid baseline directly measures this.

Why not just use degree or PageRank centrality instead of degeneracy? Degree is the symptom we are studying: RaftPPI needed degree-controlled splits precisely because models memorize hub connectivity, and the word-network work showed PageRank systematically elevates loosely-connected hubs over cohesive cores. If, on our graphs, degree ranking and $k$-core ranking agree, then degeneracy adds nothing and the audit simplifies — that agreement or disagreement is itself a reported result of Experiment 1.

Why not just iteratively strip clade directions with INLP-style classifier training? On text, one Mean Projection beat ten-to-fifteen INLP rounds on both guarding (99.4% versus 35% TPR-GAP reduction) and neighbor preservation (8 versus 43 of 120 neighbors altered) (Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection). Since our entire use case is neighbor-based retrieval, neighbor preservation is the binding constraint, so we adopt the single projection; we will still run INLP once on one benchmark as a check, since the text result does not guarantee the protein setting.

Why not just recalibrate scores against random-mutant backgrounds, as the fitness-bias paper does? Percentile recalibration (Removing bias in sequence models of protein fitness) corrects a scalar likelihood per queried protein and requires generating and scoring thousands of synthetic mutants at each mutational depth for every target; it leaves the embedding space — and hence every retrieval index, $k$-NN graph, and ANN search built on it — untouched. Our correction is computed once from clade means and repairs the index itself. The two are complementary, not rivals, for likelihood scoring; they are rivals only for retrieval, where recalibration does not operate.

Why not just retrain on a balanced database, as AbLang-2 does? AbLang-2 needed focal loss, paired heavy-light chain training, and segment masking to cut non-germline perplexity from roughly 25–30 to 10–12 — a full pretraining campaign per model and per bias type. A post-hoc projection applies to any frozen model, any database, and any newly identified skew in minutes. It is a genuine threat that retraining-level fixes reach parts of the bias (token-level generation) that pooled-embedding projection cannot; the antibody experiment is framed as a probe of how far the geometric fix goes, not as a claim to match AbLang-2 on generation.

Decisive Checks

The proposal lives or dies on three measurements. First, a linear probe must recover clade identity well above chance in raw embeddings and drop to chance after one projection; if clade identity is irreducibly non-linear — a real possibility given that contextualized text models already broke the linear norm-frequency law (On the Curious Case of l2 norm of Sense Embeddings) — the single-projection hypothesis fails and the honest output is a negative result with a characterization of where the bias actually lives. Second, hubness in the $k$-NN graph must drop in over-represented clades specifically, and $k$-core purity with respect to CATH and Pfam labels must exceed degree-ranking purity. Third, retrieval for under-represented clades must improve without overall regression against the 57.5% and 62.0% CATH20 numbers and the 0.557 Pfam20 hybrid. If all three hold, the fairness toolkit gains a biological application and every deployed PLM-based search index gains a cheap repair; if only the audit holds, we still deliver the first measurement of how training-database skews shape the topology of protein embedding spaces — a measurement that, on the evidence assembled here, nobody has made.