Teaching Language Models New Pronouns by Distilling from the Name They Replace

Language models misgender people who use neopronouns such as xe/xem or ze/zir. Part of the cause is mechanical: these words are rare in training text, so the tokenizer splits them into meaningless fragments, and the model never learns that they refer to a single person. Current repairs add the pronoun as one new vocabulary item and give it a starting vector equal to the average of he and she. That places the new pronoun between the two binary pronouns, which is a strange default for a word whose purpose is to sit outside that pair. We propose a different way to learn the new vector. The idea comes from computational chemistry, where models are trained with 3D molecular structure that will not be available when the model is used. Our training-time "structure" is the explicit referent. We build training pairs in which a teacher copy of the model reads the sentence with the person's name written in place of the pronoun, for example "Alex said Alex was tired". A student copy reads the sentence with the neopronoun, "Alex said xe was tired". Only the new pronoun's vector is trained, and the objective is that the student's predictions after the pronoun match the teacher's.

Put in two sentences: the name is the best available description of what a personal pronoun should do, since it is singular, human, and tied to one referent. So we learn the neopronoun's vector by making it act like the name, instead of copying the geometry of he and she. The name is present only during training. When the model is deployed, it handles the neopronoun alone, which is exactly the kind of structural information that is available at training time and missing at test time.

Core Claim

A neopronoun's vector learned by referent distillation will produce more consistent pronoun use than the binary-average initialization of Pronoun Tokenization Parity. The advantage should be largest where the average-of-he-and-she starting point is most misleading. That includes verb agreement, avoiding spurious determiners ("the xe"), long-range consistency in generated text, and not picking up the negative associations that Dev et al. found for non-binary terms in BERT.

A plain name-embedding initialization would not deliver the same benefit. The reason is that the teacher signal carries information that no single static vector contains: how the model's downstream computation behaves once a specific singular human referent has been named.

Where This Sits

The problem is set by the tokenization-parity study. It showed that byte-pair encoding breaks neopronouns into pieces. It then added them as whole tokens, trained only the embedding table, and initialized each new vector as the average of binary pronoun vectors. This raised neopronoun consistency on the MISGENDERED benchmark from 14.1% to 58.4% in a 70M-parameter Pythia model, and a 160M model with the fix matched a 1.4B model without it. We keep that paper's whole setup: atomic tokens, frozen transformer, and lexical-only training. We replace only its initialization and training signal. That makes it both the method we build on and the baseline we must beat.

The move of distilling structural information into a model that will not see that structure comes from DISTAL. There, a composition-only student learns to predict the internal embeddings of a 3D-aware teacher. We borrow the recipe, not the domain. DISTAL also warns that some properties could not be recovered without structure. We take that as a constraint and look for the analogous ceiling in text. Some behaviors may need the referent to be visible in the input, and trained embeddings may not recover them. Long-distance coreference across many intervening people is the obvious candidate.

Transformer-M supports the related idea that training with an optional rich channel regularizes a model that later runs without that channel. PepMLM shows that masked reconstruction from surrounding sequence context can stand in for missing structure. Our distillation is a context-conditioned version of the same idea.

Two papers explain why binary tools fail here. Lauscher et al. debias with adapters trained on gender-swapped text (swapping he for she and so on). They state that this swapping is inherently a binary toggle and breaks grammar when extended to neutral forms. Mouli et al. prove that augmentation which guesses only the single most likely context gives weaker invariance than covering the full range of contexts. That result shapes our data design: the teacher/student pairs are drawn from varied real contexts, not a single template.

Research Program

We construct pairs from WIKIBIOS biographies, the corpus the tokenization-parity study used, plus synthetic declared-pronoun passages. Each passage names a person and then refers to them with pronouns. Every pronoun mention of the target person is rewritten twice. The teacher version uses the person's name, with the name's case adjusted, for example "Alex's" for the possessive. The student version uses the declared neopronoun in the correct grammatical form (xe/xem/xyr/xemself).

For each pair we freeze all weights except the new pronoun vectors. We then minimize

$$ \mathcal{L} = \sum_{t \in T} \mathrm{KL}\big(p_{\text{teacher}}(\cdot \mid x^{\text{name}}_{<t}) \,\|\, p_{\text{student}}(\cdot \mid x^{\text{pron}}_{<t})\big) + \lambda \, \mathcal{L}_{\text{LM}}. $$

Here $T$ is the set of positions after each rewritten mention, aligned between the two versions because the name and the pronoun may differ in token count. The two $p$ terms are the next-token distributions of the teacher and the student. $\mathcal{L}_{\text{LM}}$ is ordinary next-token loss on the student text, so the student also learns to produce the pronoun at the right places. The weight $\lambda$ balances the two terms.

The KL term transfers exactly what we want: singular agreement ("xe is"), person-hood, and continued tracking of the referent. It excludes the one thing the name carries that the pronoun should not: the name's own identity-specific associations. To keep those out, we average the teacher signal over many different names per template.

We then analyze where the learned vectors end up. The first question is how much of each vector lies in the he–she direction. The second is whether its nearest neighbors are people or, as Dev et al. found for ze in GloVe, foreign-looking strings. The third is whether it inherits the negative-adjective associations they report.

Baselines

The first reference point does nothing: the unmodified Pythia model with its standard tokenizer. On the 70M model this gives the 14.1% MISGENDERED neopronoun consistency and 85% adversarial injection error reported in the tokenization-parity study.

The strongest existing method is that study's full recipe: atomic tokens, binary-average initialization, and lexical fine-tuning with gender-swapped WIKIBIOS text. It reports 58.4% consistency and 27% injection error at 70M. It is the strongest competitor because it already fixes fragmentation, which accounts for most of the gap. We rerun it ourselves on the same Pythia checkpoints (70M, 160M, 410M, 1.4B), the same WIKIBIOS passages, the same number of updated parameters, and the same number of optimizer steps and hyperparameter search budget. Beating it means higher MISGENDERED neopronoun consistency and lower injection error at every model size, with seed variance reported. It also means a clear margin on the long-form checks described below, where we expect the averaged initialization to hurt most.

Three ablations isolate the key ingredient, which is the teacher signal. The first keeps our training data but drops the KL term, leaving only language-model loss. The second keeps the KL term but uses a single fixed name, which tests whether name averaging matters. The third keeps the parity recipe but initializes from a different static vector: the singular they embedding, or the mean embedding of common first names.

Why not just…?

Why not just initialize from a better vector, such as a name embedding or they? This is the most serious threat, and it may account for much of any gain. A static initialization fixes where the vector starts. It does not tell the model how downstream layers should treat a singular referent. Singular they is a questionable anchor, because Dev et al. found BERT resolves it correctly only 52.1% of the time and defaults to plural. The third ablation above settles the question. If a name-mean initialization plus lexical fine-tuning matches referent distillation, then the contribution is really a better initialization, and we will report it that way.

Why not just scale the model or declare pronouns in the prompt? MISGENDERED already includes an explicit pronoun declaration, and the baseline still scores 14.1%. The tokenization-parity study shows that a 160M model with the fix matches a 1.4B model without it, so scaling alone is an expensive substitute. We include the 1.4B model without any fix as a scaling reference.

Why not just rewrite neopronouns to singular they at input and back at output? This does not handle generation, where the model has to choose the right pronoun on its own. It also passes everything through a pronoun that models often read as plural. We test it as a preprocessing baseline on the generation tasks.

Proposed Experiments

The main test is MISGENDERED, covering neopronoun consistency and adversarial injection, across the four Pythia sizes. General ability is checked on Winogrande and ARC-Challenge. No reported result was found on Lacuna for exact scores on those two benchmarks under the parity recipe, so we will measure the baseline ourselves.

Because Anthis shows that template tests barely track realistic use (average Spearman ρ = 0.12), the decisive check is long-form. We will adapt the bedtime-story and user-persona generation tasks from that work so that the protagonist declares a neopronoun. On outputs of up to 1,000 tokens, we then measure three things: the fraction of pronoun references that stay correct, spurious-determiner errors ("the xe"), and verb-agreement errors.

The failure-mode test builds on DISTAL's ceiling: passages with two or three named people, where a trained embedding may not be enough to track the right referent. If referent distillation beats the parity recipe on long-form consistency but not on multi-referent passages, that locates the limit of training-time-only structure. If it fails to beat the name-initialization ablation, the distillation claim fails.