Machine translation systems that translate English into languages with grammatical gender (Spanish, Italian, German) tend to choose masculine word forms by default, even when the sentence says otherwise. They write the Italian equivalent of "my doctor is pregnant" with a masculine dottore. This proposal treats that failure as a long-distance-dependency problem, close in shape to one that protein modeling has already worked on. In proteins, two amino acids far apart in the chain can sit next to each other once the chain folds. Models that read only the chain miss those contacts. A recent fix teaches a sequence-only protein model, during a short second training phase, to predict structural facts it cannot see. In translation, the word that fixes a noun's gender ("she", "pregnant", "father") can be many tokens away in the linear string yet only one or two hops away in the sentence's grammatical structure. We propose a short post-training phase in which a sequence-only translation model learns to point, from each gendered output word, back to the source word that controls its gender. At test time the model needs no parse.
Two things are new. The first is a diagnostic: we test whether masculine-default errors grow with linear token distance to the gender cue or with grammatical distance. Existing benchmarks mix the two. The second is a training method that copies the structure-prediction objective from protein language models but replaces 3D geometry with agreement links. We test it against the cheapest serious alternative, gender-balanced data augmentation, on matched data.
Core Claim
Interpretability work shows that misgendered translations happen when the model gives near-zero attribution to the distant pronoun. A Tale of Pronouns measured this with Integrated Gradients on Flan-T5 and mT0 over WinoMT. Architecture work finds that shared multilingual encoders show "tunnel vision", attending along the diagonal to the local occupation word, while language-specific encoders attend to non-local cues (Interpreting Gender Bias in NMT: Multilingual Architecture Matters). Both papers describe a model that ignores a controller far away in the string, which is the same pattern as protein models that cannot see long-range contacts.
Our claim has two parts. First, part of the masculine default is a failure to route information across linear distance, separate from frequency priors. Second, an auxiliary objective that forces the model's internal states to locate the gender controller will cut errors specifically where the controller is far away. Counterfactual rebalancing of the training data should not do this, because it corrects how often each gender appears but gives no signal about where to look.
What We Borrow and Where We Differ
The method comes from Structure-Aligned Protein Language Model. That paper post-trains ESM2 with two objectives. One contrastively aligns residue embeddings to a frozen geometric graph network (GearNet). The other predicts discrete structural tokens from sequence states. It also restricts training to residues with high "excess loss", meaning the current model fails there but a reference model succeeds. It reports a 59.1% relative gain in long-range contact Precision@L/5 on CASP16 with under 6 hours on 8 H100s, and it keeps sequence-only inference. We reuse all three ingredients, but the structure we target is the agreement and coreference link rather than spatial contact.
GearNet's own finding, that an explicitly relational model trained on 805k structures matches ESM-1b trained on 24M sequences, motivates the bet that explicit structure is a denser signal than more raw text. Our design does not depend on that finding transferring to language.
Two results set the constraints. Rao et al. and Chen et al. show that masked-token pretraining at scale does not recover structure, which argues against "just scale". Is there Gender Bias in Dependency Parsing? shows that apparent gender effects in parsing were really dependency-length effects. That is a warning that any gender result here has to control for distance, so we build distance control into the evaluation.
Our method differs from the prompting fix in A Tale of Pronouns in three ways: it changes weights rather than prompts, it needs no gradient access at inference, and it targets distance directly.
Research Program
Supervision. We take English-to-{Spanish, Italian, German, French} parallel data, including Europarl. Europarl is heavily male-skewed: about 30% female utterances per Gender Bias in Machine Translation, and under 20% feminine terms in EN-DE/EN-FR per Does Context Help Mitigate Gender Bias in NMT?. Its skew is part of the problem. We run an off-the-shelf dependency parser and coreference resolver on the source side and word-align the target side. Every target token carrying gender morphology then receives a silver label $c$: the source position of the token that determines its gender, or a "none" label when the source leaves gender ambiguous. Parsers are used only to make training labels.
Objective. The model is translated as usual. A small pointer head reads the decoder state $h_t$ at each gendered target token $t$ and scores source positions. We minimize
$$\mathcal{L} = \mathcal{L}_{\text{MT}} + \lambda \, \mathbb{E}_t\left[-\log p(c_t \mid h_t)\right],$$
where $\mathcal{L}_{\text{MT}}$ is the standard translation loss, $p(c_t \mid h_t)$ is the pointer distribution over source positions plus "none", and $\lambda$ is a scalar weight. This plays the role of SaESM2's structural-token prediction.
A second, optional term mirrors SaESM2's latent alignment. It contrastively aligns encoder token states with the node states of a frozen graph encoder run over the dependency-plus-coreference graph, and we ablate it separately.
Excess-loss curation. The reference model is a parse-aware translator that receives the graph at input. We restrict the link loss to the gendered tokens where the base model's gender-token loss is high but the reference model's is low. This is the automated analogue of the hand-picked "hard" examples in A Tale of Pronouns.
Models. We use mT0 and Flan-T5, so that results compare directly with A Tale of Pronouns, and M2M-100, the dedicated multilingual system evaluated in Investigating Failures of Automatic Translation in the Case of Unambiguous Gender.
Diagnostic set. We extend the phrase-structure grammar approach of Renduchintala and Williams with a two-by-two stimulus design that separates the two kinds of distance. Inserting adverbials and parentheticals raises linear distance between controller and occupation noun while holding the dependency path fixed. Embedding the noun inside clauses raises path length while holding token distance roughly fixed. Each item has masculine and feminine counterfactuals and pro- and anti-stereotypical occupations.
Baselines
Unmodified models. The naive reference point is each model fine-tuned with $\mathcal{L}_{\text{MT}}$ alone on exactly the same parallel data and step count. This isolates the effect of extra training from the effect of the link objective. Reported context for how weak these systems are: M2M-100 fails to reach 70% accuracy on the unambiguous 20-language benchmark (Investigating Failures), and systems score 50–61% on MT-GenEval's inter-sentential subset versus over 95% for human translators (MT-GenEval).
Interpretability-guided few-shot prompting. This is the strongest existing method on the same question (A Tale of Pronouns). It reports that WinoMT English-to-Spanish gender gap $\Delta_G$ fell from 7.2 to 2.1, with +7.1 accuracy, using four hard demonstrations selected by low pronoun attribution, on Flan-T5/mT0. We rerun it with the authors' selection procedure on the same model checkpoints. To beat it, our method must reach $\Delta_G$ at or below 2.1 on English-to-Spanish WinoMT. It must also beat the prompting method on the far-linear-distance cells of the diagnostic set, where we expect four demonstrations to help least.
Counterfactual data augmentation. We fine-tune on the same parallel data, adding gender-swapped counterfactual copies of sentences, with the same number of updates and the same hyperparameter search budget as our method. No reported result found on Lacuna for counterfactual augmentation on these benchmarks, so we produce the number ourselves.
Linear context concatenation. This is the 2-to-1 sentence concatenation of Does Context Help Mitigate Gender Bias in NMT?. That paper reports EN-DE feminine generative accuracy rising from about 5% to about 23%, with degradation in male-skewed domains. We run it as the baseline for MT-GenEval's contextual subset.
Parse at inference (reference ceiling). The parse-aware reference translator, which takes the graph as input, serves as a ceiling. It tells us how much of the gain from explicit structure the sequence-only model recovers. This mirrors the GearNet-versus-SaESM2 trade-off.
Ablations. These isolate the key ingredient: link loss without curation, curation without link loss (fine-tuning only on the hard tokens), latent alignment alone, and a link loss whose labels are shuffled to random source positions. The shuffled-label ablation rules out a pure regularization effect.
Why not just…?
Why not just augment the data with gender-swapped counterfactuals? This is the most serious threat. Augmentation directly attacks the frequency prior behind the "masculine sink" described by Mohammadi et al., and it costs nothing at inference. Our hypothesis is that it closes the gap where the controller is close by and leaves it open where the controller is far, because balanced counts say nothing about where to look.
The matched comparison settles this, analyzed per cell of the distance design. If augmentation equals our method in the far-linear-distance cells, the routing hypothesis is wrong and we would report augmentation as the answer. We also test the two combined, because they may help in complementary ways.
Why not just feed the model longer context? Does Context Help Mitigate Gender Bias in NMT? found that concatenation gains concentrate on already-stereotypically-female occupations. It also found that in parliamentary text, extra context lowers feminine accuracy, because the model reads domain statistics rather than the coreference chain. Our objective supervises the chain itself. We test whether it avoids this amplification on the Europarl-domain subset.
Why not just give the model the parse at test time? This works only when parsing and coreference resolution are reliable. It also adds a pipeline dependency for every language and genre, which is the same input burden that structure-based protein models carry. We include it as a ceiling rather than a competitor. If the sequence-only model recovers most of its gain, the extra dependency is not worth carrying.
Why not just scale or prompt? Unambiguous-gender failures persist in large multilingual systems (Investigating Failures) and in frontier LLMs on late binding (MiTTenS). Lexical-constraint prompting reduces but does not remove bias (Beyond Binary Gender). Constraint prompting also needs to be told the gender at test time. We still include the strongest prompting method as a baseline.
Proposed Experiments
Is there a gap to close? The first experiment needs no training. We fit gender accuracy on the diagonstic set as a function of linear distance and of dependency-path length, for each base model. If error depends on path length but not on linear distance once path length is held fixed, the analogy to the protein sequence-structure gap fails. In that case we would redirect the work toward the syntactic difficulty itself. This experiment is the gate for everything else.
Main evaluation. We measure accuracy, $\Delta_G$, and $\Delta_S$ on WinoMT across the four target languages. We report accuracy on MT-GenEval's contextual subset and its masculine-feminine BLEU gap; that paper reports a gap of up to 13.0 BLEU on EN-DE. We report accuracy on the 20-language unambiguous benchmark, restricted to our target languages. We also report late-binding accuracy on MiTTenS. MiTTenS translates into English, so it tests whether pointer training on the encoder side transfers when the source language carries the gender.
General translation quality. We report BLEU and COMET on held-out general-domain test data, because the protein precedent itself showed small losses in sequence-modeling fidelity after alignment.
Decisive check. The method has to show a larger gain in the far-linear-distance cells than in the near cells, relative to matched augmentation. The shuffled-label ablation has to show no such gain. If both hold, we have evidence that the masculine default is partly a routing failure that structure supervision repairs. If not, the result still tells the field that rebalancing frequency is enough.
References
- A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation
- Interpreting Gender Bias in Neural Machine Translation: Multilingual Architecture Matters
- Structure-Aligned Protein Language Model
- Protein Representation Learning by Geometric Structure Pretraining
- Evaluating Protein Transfer Learning with TAPE
- TourSynbio: A Multi-Modal Large Model and Agent Framework to Bridge Text and Protein Sequences for Protein Engineering
- Is there Gender Bias in Dependency Parsing? Revisiting “Women’s Syntactic Resilience”
- Gender Bias in Machine Translation
- Does Context Help Mitigate Gender Bias in Neural Machine Translation?
- Investigating Failures of Automatic Translationin the Case of Unambiguous Gender
- MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation
- Identifying Gender Stereotypes and Biases in Automated Translation From English to Italian Using Similarity Networks
- MiTTenS: A Dataset for Evaluating Gender Mistranslation
- Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words