Layer-Localized Removal of Murine Identity in Antibody Language Models

Therapeutic antibodies are usually born in mice: a mouse antibody binds the disease target well, but a patient's immune system recognizes the mouse-derived parts of the protein as foreign and raises anti-drug antibodies (ADAs) that can clear the drug from the body or cause toxic reactions. Humanization — rewriting the non-binding scaffold of the antibody so it looks like a normal human antibody while keeping the target-binding loops intact — is therefore a central design problem, and language models trained on natural antibody repertoires increasingly do the rewriting. The catch is that these models carry their own unwanted preferences: they over-predict germline residues 85–95% of the time even on highly mutated sequences (Addressing the antibody germline bias and its effect on language models for improved antibody design), and current fixes are blunt — retrain the whole network with a reweighted loss, or wrap inference in an external auditing loop that re-masks suspicious residues.

Meanwhile, research on architectural bias in language models has found something sharper: unwanted preferences are often localized. A model that confuses high word overlap with genuine semantic equivalence shows this confusion peaking in specific middle layers, and a small projection network inserted at layer 2 of T5-small cut that lexical bias from 46.80% to 6.40% (Resolving Lexical Bias in Model Editing). Hidden teacher preferences transfer to student models during distillation through a single early layer (Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer). Nobody has asked whether the same layer-localization holds for the most consequential bias in antibody models: the retained identity of the murine parental sequence that humanization is supposed to erase.

Core Claim

I propose that "this sequence is mouse" is not uniformly encoded across the layers of an antibody language model, and that the layers carrying species identity can be found, causally verified, and selectively scrubbed during generation. If species identity concentrates in identifiable middle layers while structural constraints — the framework residues, especially the Vernier zone beneath the binding loops, that hold the antibody's shape together (Title) — are encoded earlier, then a lightweight, layer-targeted intervention should humanize at least as well as whole-model approaches while preserving more of the parental sequence. The hypothesis is testable, and its confirmation would turn humanization from an output-level guessing game into a targeted representational edit.

Research Program

The program has three parts, each reusing a technique proven in the NLP bias literature and each grounded in an existing antibody resource.

First, a layer-wise audit. For each frozen antibody model — AbLang-2, the masked language model at the heart of the Product-of-Experts humanization system (Generative Humanization for Therapeutic Antibodies), and the discrete-diffusion backbone HuDiff-Ab refined by AIR (AIR: Inference-Time Refinement for Discrete-Diffusion Antibody Humanization) — I will train linear probes, one per layer, to decode three things from the hidden representations: species (human versus mouse versus camelid, all abundant in the Observed Antibody Space, a database of over 600 million antibody sequences from 53 studies (Observed Antibody Space: A Resource for Data Mining Next-Generation Sequencing of Antibody Repertoires)); the AbNatiV nativeness score, whose humanness component correlates negatively (R = −0.49) with clinical ADA incidence across 216 therapeutics (Assessing antibody and nanobody nativeness for hit selection and humanization with AbNatiV); and germline gene family. The prediction, imported from the lexical-bias finding that superficial signals peak in middle layers (Resolving Lexical Bias in Model Editing), is a species-decodability spike in middle layers with structural information decodable earlier.

Second, causal verification. Correlation is not enough: following the activation-patching methodology used to show that fine-tuning a single early layer suffices to transfer hidden biases in distillation (Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer), I will intervene on the identified layer during masked reconstruction of therapeutic sequences — projecting out the species-decodable direction — and measure whether murine framework residues in the reconstruction disappear while Vernier-adjacent residues survive. This is the decisive mechanistic test: if scrubbing the middle layer does not remove murine residues, the localization story is wrong and the project stops there, having still produced the first bias-atlas of antibody models.

Third, the intervention and a controlled architectural comparison. The intervention is a PENME-style lightweight projection network, inserted at the identified bias-peak layer and trained with a contrastive objective: drive species-probe decodability to chance while preserving nativeness and the affinity and thermostability oracles already built in the Product-of-Experts work (Generative Humanization for Therapeutic Antibodies). Alongside this, I will train four small matched-parameter backbones — a masked encoder, a causal decoder, a discrete-diffusion model, and a 1D convolutional network in the style of Humatch (Humatch - fast, gene-specific joint humanisation of antibody heavy and light chains) — on the identical 1.26 million paired-sequence OAS partition curated for AbLang-2 (Addressing the antibody germline bias and its effect on language models for improved antibody design). This settles a live tension: AbLang-2's authors treat germline bias as a data-distribution artifact fixed by loss reweighting, whereas multilingual StereoSet probing showed identical text data producing opposite bias profiles in encoder versus decoder architectures, with English BERT at stereotype scores of roughly 58–60 where 50 is neutral and multilingual GPT-2 dipping below it (How Different Is Stereotypical Bias Across Languages?). If the four backbones show the same layer profile, the bias is data-driven; if they diverge, architecture is a first-order cause and model choice becomes a humanization design decision.

Baselines

The naive reference point is doing nothing: the unedited murine parental sequences from Humab25, the gold-standard benchmark of 25 clinical antibody pairs with known murine precursors and lab-humanized end states. No reported result found on Lacuna for the humanness score of the unedited parental sequences, so I will measure it directly; the lab-humanized end states are the target, with a lab-standard T20 humanness score of 13.56 reported on Humab25 by the AIR paper.

The strongest existing competitor is AIR-Sequential, the inference-time wrapper around the HuDiff-Ab discrete-diffusion model that audits generated residues using confidence and the external AbNatiV oracle, re-masks flagged positions, and resamples them under proximity-based Vernier zone protection (AIR: Inference-Time Refinement for Discrete-Diffusion Antibody Humanization). It is the strongest because it is the only method that both matches lab humanness on Humab25 (T20 of 13.34 against the lab's 13.56) and retains roughly 80.4% of the parental sequence without any retraining. To make the comparison fair, I will run AIR and my layer-targeted scrubber on the same frozen HuDiff-Ab backbone, with identical sampling budgets, the same AbNatiV oracle access, and the same wall-clock allowance, and report T20 humanness, total parental preservation, and Vernier-zone preservation on identical Humab25 splits. Beating AIR means matching or exceeding its T20 of 13.34 while raising total and Vernier preservation by at least three percentage points, or matching its 80.4% preservation at a strictly higher T20.

Two further method baselines calibrate the field. Hu-mAb, a random-forest scorer over repertoire data, suggested on average 59% of the mutations made experimentally and achieved adjusted overlap ratios of 77% for heavy and 85% for light chains on Humab25, at about 1093 seconds per antibody (Humanization of antibodies using a machine learning approach on large-scale repertoire data). Humatch's three-CNN system matched 77% of heavy-chain and 82% of light-chain experimental mutations in 33 seconds per paired antibody (Humatch - fast, gene-specific joint humanisation of antibody heavy and light chains); both will be run on the same Humab25 pairs for mutation-match and runtime comparison. Finally, two ablations isolate the proposal's key ingredient: the same projection network inserted at a deliberately wrong (late) layer, and a whole-model fine-tune with the same species-adversarial objective but no layer targeting. If the wrong-layer and whole-model variants match the targeted one, layer localization is doing no work and the claim collapses to "adversarial training helps," which would be a much smaller result.

Why not just…?

Why not just retrain with focal loss, as AbLang-2 already did? The AbLang-2 authors cut non-germline residue perplexity from roughly 25–30 down to 10–12 using focal loss, segment masking, and a 6-to-12-layer scale-up, treating the bias as a property of the data distribution (Addressing the antibody germline bias and its effect on language models for improved antibody design). That fix targets germline over-prediction generally, not retained murine identity in a specific parental antibody, and it requires full retraining at scale for every base model. My approach is post-hoc on frozen models and tests the stronger architectural claim. The matched-backbone experiment is designed precisely to arbitrate between the data explanation and the architecture explanation; if all four backbones show identical bias on identical data, the AbLang-2 view wins and I will say so.

Why not just use AIR as it stands? This is the real threat, and I want to be honest about it: AIR already reaches lab-level humanness without touching the model. But AIR is blind to mechanism — it cannot say why the diffusion model proposes murine residues, it depends on an external oracle and hand-set Vernier proximity rules, and the AIR paper itself lists the mechanistic reason for premature diffusion commitment as an open question. My proposal predicts that a layer-targeted scrub achieves comparable humanness with higher Vernier preservation and no external oracle dependency. The head-to-head experiment settles it either way, and even a loss would leave the layer atlas as a diagnostic contribution that could improve AIR's re-masking policy.

Why not just generate many candidates and filter them with AbNatiV? AbNatiV is a strong scorer (precision-recall AUC of 0.996 separating human from mouse heavy chains), but it does not generate; filtering candidates from a biased generator inherits the generator's blind spots, and its ADA correlation, while significant, is R = −0.49 across 216 clinical antibodies — a screening funnel, not a fix (Assessing antibody and nanobody nativeness for hit selection and humanization with AbNatiV). The Product-of-Experts work showed why this matters empirically: deterministic argmax optimization from a biased prior produced low-diversity candidates that frequently failed binding entirely in wet-lab tests, while guided probabilistic sampling from the same prior yielded orders of magnitude more viable sequences (Generative Humanization for Therapeutic Antibodies). Filtering cannot repair the prior; scrubbing the bias at its representational source can. AbNatiV still appears in my pipeline as an evaluation metric, which is the role it is actually good at.

Proposed Experiments and Decisive Checks

The evaluation spine is Humab25: T20 humanness score (target at or above AIR's 13.34, aspirationally the lab standard of 13.56), total parental preservation (target above 80.4% at matched humanness), and a new Vernier-zone preservation metric computed from AHo numbering. Because 25 antibodies limit statistical power — a limitation the AIR authors themselves flag — I will replicate the headline comparison on HuAb348, the larger benchmark named in the AIR paper's future work; no reported result found on Lacuna for HuAb348, so I will establish its reference numbers under AIR and the naive parental baseline as part of the study. Downstream immunogenic relevance is checked by scoring all generated candidates with AbNatiV against the Marks clinical ADA set of 216 therapeutics, asking whether layer-scrubbed designs shift the score distribution toward the low-ADA region, with Humatch's reported cutoff behavior (removing 89% of high-ADA, 59% of medium-ADA, and 23% of low-ADA therapeutics at a 0.01 threshold) as the reference operating point (Humatch - fast, gene-specific joint humanisation of antibody heavy and light chains). Success is defined crisply: a reproducible layer-resolved species-bias map, a causal demonstration that scrubbing the identified layer removes murine framework residues without collapsing Vernier residues, and a humanization result on Humab25 and HuAb348 that beats AIR's preservation–humanness frontier. Failure modes are equally informative — a flat layer profile would be the first direct evidence that germline and species biases in protein models are qualitatively unlike the lexical and subliminal biases of text models, which would itself redirect both literatures.