Back-translation is a standard way to train a translation system when human-translated sentence pairs are scarce. A second model translates monolingual text in the target language back into the source language, and the resulting synthetic pairs are used as extra training data. In the iterative version, the two directions take turns producing training data for each other over several rounds. This note argues that the loop has a specific and measurable fairness failure. When the target sentence says "the doctor (feminine) asked for her notes," the backward model often writes an English source with "his notes." The forward model then sees a training pair whose source gender cue contradicts its target. These contradictions are not spread evenly. They fall mostly on anti-stereotypical sentences, which are exactly the cases where the backward model's stereotyped default is wrong. The forward model therefore learns that explicit gender cues are unreliable in the very cases where they matter, and it falls back on occupational stereotypes. Each round feeds those stereotyped outputs into the next round's data.
The opportunity is threefold: to measure this "cue corruption" directly in the synthetic data, to show that it explains how bias grows across rounds better than the demographic skew of the monolingual corpus does, and to repair it cheaply. The repair edits the synthetic source so that it agrees with the gender already marked in the real target. Dropping the bad pairs is not enough. We predict that dropping them throws away a disproportionate share of anti-stereotypical examples and makes the training data more stereotyped, not less.
Core Claim
The claim is that iterative back-translation amplifies gender bias through group-disparate source-side noise, not mainly through skewed data. Synthetic sources contradict the target's gender far more often for anti-stereotypical entities than for stereotypical ones. Under that condition, the stereotype gap grows round over round even while BLEU improves. A cue-preserving repair of the synthetic source should flatten that growth at no BLEU cost. Two weaker interventions should fail to match it: tagging synthetic data, and filtering out mismatched pairs.
This mechanism is borrowed from theory. An Effective Theory of Bias Amplification shows, in high-dimensional ridge regression, that disparity between groups can be amplified even on perfectly balanced 50/50 data when the groups differ in label-noise rates. That result is the reason we expect the disparity in corruption rates, not the overall noise level, to drive amplification. Back-translation is a natural real-world generator of exactly this kind of group-dependent noise, which linear theory cannot model directly. We test the prediction empirically rather than claiming the theory transfers.
Relation to Prior Work
Two papers motivate the proposal. Bias Amplification: Large Language Models as Increasingly Biased Media fine-tunes GPT-2 on its own generations for ten rounds on political news. It finds that political bias grows while text quality stays stable, and that mixing real data back in ("Preservation") stops quality collapse but not bias growth. That work is monolingual and self-consuming. We study a two-model, cross-lingual loop where each training pair's input is synthetic but its target is real, and the bias enters through the input side. Gender bias amplification during Speed-Quality optimization in Neural Machine Translation shows that BLEU and gender accuracy come apart in translation, with only a 6–9% BLEU loss alongside a 41.3% drop in anti-stereotypical accuracy for English–German under speed optimizations. That study looks only at fixed models at inference time and does not examine training loops.
Three lines of work supply the evaluation and the comparison methods. Evaluating Gender Bias in Machine Translation defines WinoMT and its alignment-plus-morphology scoring pipeline. We reuse that pipeline both to score models and to detect cue corruption inside synthetic training pairs, which is a use it was not designed for. MT-GenEval introduces gender-filtered self-training in a single-pass setting. Its filtering idea is the obvious competitor to our repair, but it was never tested across rounds of synthetic generation. Target-Agnostic Gender-Aware Contrastive Learning (GACL) debiases using a contrastive loss over gender-balanced human bitext. It is the strongest existing debiasing method, but it depends on curated human data that low-resource back-translation setups have in short supply.
Two papers constrain the design. Tagged Back-Translation marks synthetic sources with a reserved token so the model can discount their noise, and it reaches 33.4 BLEU on English–Romanian at iteration 3. It never measured bias, and it is a serious candidate for partly solving our problem for free. FairBelief finds harmful associations at the peak of model likelihood rather than in the tails. That finding argues against decoding tweaks as a fix, although sampled back-translation still deserves a direct test. The survey Gender Bias in Machine Translation names iterative synthetic data as an unstudied route for bias. The iterative protocol itself follows Iterative Back-Translation for Neural Machine Translation.
Research Program
We would build an iterative back-translation pipeline for English↔German and English↔Spanish. These two pairs are covered by WinoMT, SimpleGEN and MT-GenEval. The pipeline follows the low-resource simulation of Iterative Back-Translation for Neural Machine Translation: Transformer-base models, 100K and 1M human sentence pairs from WMT, WMT News Crawl monolingual text, and three to four rounds.
The first component is a cue-agreement auditor that runs on every synthetic training pair. It aligns the synthetic English source to the real target using the WinoMT tooling, reads grammatical gender from the target with a morphological analyzer, and reads gender from the aligned English pronoun. It labels each pair as agreeing, contradicting, or carrying no cue. Each person entity is also labeled stereotypical or anti-stereotypical using the WinoBias occupation statistics. This gives two quantities per round: a corruption rate for stereotypical entities and one for anti-stereotypical entities.
The second component is the repair, called cue-preserving back-translation. When the auditor flags a contradiction, the system first rescores the backward model's n-best list and keeps the most probable candidate whose pronouns agree with the target. If no candidate agrees, it applies a rule-based swap of English pronouns, which form a small closed set. Pairs where the target carries no gender marking are left untouched, because forcing a cue there would invent information. The repair needs no human bitext beyond what the loop already uses.
The third component covers the bidirectional coupling. When the English→German model produces synthetic German for the reverse direction, its wrong noun genders become the reverse model's inputs. We audit both directions to see whether corruption in one round predicts corruption in the next, which would be a ratchet across the loop.
Baselines
The naive reference point is standard untagged iterative back-translation with beam search, plus the single-round model trained on human bitext only. Every other system is reported as a change relative to these.
The strongest existing debiasing method is GACL from Target-Agnostic Gender-Aware Contrastive Learning. It reports WinoMT gender accuracy rising from 61.7% to 78.2% on NLLB-200, with FLORES-200 spBLEU moving from 40.5 to 40.0. It is the strongest competitor because it directly targets context-driven gender errors without hurting translation quality. We run it fairly in two ways. In the first, GACL is applied after the loop to the final-round model, using the gender-balanced filtered subset of the same 100K or 1M human pairs. In the second, GACL fine-tuning runs inside every round. Both variants get the same compute and tuning budget as our method. Beating GACL means one of two outcomes: a lower stereotype gap on WinoMT at equal BLEU, or a matched stereotype gap obtained from less human data. If GACL plus plain back-translation matches us, we would report that the cheap repair offers no advantage when some human bitext exists, and confine our claim to the regime with very little bitext.
Tagged back-translation, following Tagged Back-Translation, is run with an identical loop, data and number of rounds. No reported bias result was found on Lacuna for tagged back-translation; its only reported figure is 33.4 BLEU on English–Romanian. Gender-filtered self-training extended to multiple rounds, following MT-GenEval, uses the same auditor but drops contradicting pairs instead of repairing them. No reported numeric result was found on Lacuna for gender-filtered self-training. We also include sampled rather than beam back-translation, and a balanced-pool baseline that equalizes gendered-entity counts per occupation in the monolingual data before each round.
One ablation isolates the key ingredient, the equalization of corruption rates across groups. A "disparity-matched" control injects pronoun flips into stereotypical pairs until both groups have the same corruption rate. Overall noise goes up, but the disparity disappears. If this control reduces amplification even though it adds noise, disparity rather than noise level is the cause.
Why not just…?
Why not just tag synthetic data and trust the model to discount it? The tag teaches the model that synthetic sources are unreliable in general. However, Bias Amplification: Large Language Models as Increasingly Biased Media found that interventions which stop quality collapse leave bias growth intact, and that the two are carried by largely separate neurons. This is a real threat, because the tag might also teach the model to ignore source cues only in tagged inputs, leaving real inputs unaffected. The tagged baseline, evaluated round by round on WinoMT and SimpleGEN, settles the question directly. If tagging alone keeps the stereotype gap flat, the repair is unnecessary and we would say so.
Why not just filter out the mismatched pairs? Filtering is cheaper and is the closest prior method. Our specific prediction is that contradictions concentrate on anti-stereotypical entities, so filtering removes them selectively and pushes the corpus toward stereotypes. The auditor measures the anti-stereotypical share of the training data before and after filtering in every round. If that share does not drop, or if filtering matches repair on the stereotype gap, the contribution shrinks to the diagnosis alone.
Why not just balance the monolingual corpus? The theory paper argues that amplification survives balanced data when noise is group-disparate, and GACL's results show that models ignore explicit cues even in clean parallel data. The balanced-pool baseline crossed with repair on or off tests this directly. Our claim predicts that balancing alone barely changes the stereotype gap's growth, while repair works on both skewed and balanced pools.
Why not just sample instead of beam-search during back-translation? Sampling produces the minority gender roughly in proportion to the model's own probability. That could reduce corruption, although FairBelief suggests the stereotyped choice sits at the peak of the distribution. Because sampled back-translation is common practice, it is run as a full baseline, and its corruption rates per group are reported.
Proposed Experiments and Decisive Checks
Every system is evaluated after every round. Translation quality uses BLEU and COMET on WMT newstest. Gender behavior uses WinoMT accuracy, gender gap and stereotype gap; SimpleGEN anti-stereotypical accuracy split by male and female contexts; and MT-GenEval contextual accuracy and quality gap. The corpus auditor's per-group corruption rates are reported alongside.
The first decisive check is descriptive. It asks whether BLEU rises while the WinoMT stereotype gap and SimpleGEN female-context errors grow across rounds of plain back-translation. If they do not, the premise fails, and we report a null result on an important practice.
The second check is mechanistic. We regress round-over-round growth in the stereotype gap on the difference in corruption rates between groups and, separately, on the gender skew of the monolingual pool. The claim requires the corruption disparity to be the stronger predictor.
The third check is causal. Starting from a repaired corpus, we inject controlled pronoun flips at fixed overall rates while varying how they split between groups. We expect a dose-response: amplification should rise with disparity at constant total noise, and stay flat under the disparity-matched control.
The fourth check compares interventions. Repair should beat plain, tagged, filtered, sampled and balanced-pool back-translation on the stereotype gap with BLEU within noise, and should be competitive with GACL when human bitext is limited to 100K pairs.
The study has limits. All three benchmarks encode binary gender only. The auditor inherits alignment errors, so we will hand-check a stratified sample of several hundred audited pairs per language and report the auditor's precision. Rule-based pronoun swaps can occasionally produce awkward English, so we will report how often the n-best repair succeeds compared with the rule fallback.
References
- An Effective Theory of Bias Amplification
- Bias Amplification: Large Language Models as Increasingly Biased Media
- Gender bias amplification during Speed-Quality optimization in Neural Machine Translation
- Evaluating Gender Bias in Machine Translation
- MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation
- Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation
- Tagged Back-Translation
- FairBelief - Assessing Harmful Beliefs in Language Models
- Gender Bias in Machine Translation
- Iterative Back-Translation for Neural Machine Translation