Abstract: Machine translation systems with inadequate document understanding can make errors when translating dropped or neutral pronouns into languages with gendered pronouns (e.g., English). Predicting the underlying gender of these pronouns is difficult since it is not marked textually and must instead be inferred from coreferent mentions in the context. We propose a novel cross-lingual pivoting technique for automatically producing high-quality gender labels, and show that this data can be used to fine-tune a BERT classifier with 92% F1 for Spanish dropped feminine pronouns, compared with 30-51% for neural machine translation models and 54-71% for a non-fine-tuned BERT model. We augment a neural machine translation model with labels from our classifier to improve pronoun translation, while still having parallelizable translation models that translate a sentence at a time.
Introduction
Machine translation often stumbles when translating "dropped" or neutral pronouns (like the Spanish possessive su) into gendered languages like English, frequently defaulting to masculine pronouns. This paper introduces a clever "cross-lingual pivoting" technique that uses Wikipedia to automatically generate massive, high-quality gender-labeled datasets. These datasets are then used to fine-tune a BERT classifier, significantly improving translation accuracy by providing the necessary gender context.
The Challenge of Ambiguous Pronouns
In many languages, such as Spanish, subject pronouns can be omitted (a phenomenon known as "pro-drop") or are gender-neutral. For instance, the Spanish possessive su can translate to his, her, or their depending on the context. When translating these into a language with strict gender marking like English, models often fail because the necessary gender cues are "extra-sentential"—meaning they exist in the surrounding paragraph, not the sentence being translated.
Standard Neural Machine Translation (NMT) models typically translate one sentence at a time. Without the broader context, these models often fall back on corpus biases, frequently translating ambiguous Spanish pronouns into the English masculine "he" or "his," even when the context clearly refers to a female subject.
Methodology: Cross-Lingual Pivoting
The authors propose a three-stage pipeline to solve this without requiring expensive human labeling:
- Cross-Lingual Pivoting: They exploit the fact that Wikipedia often has pages on the same subject in multiple languages. By aligning Spanish and English Wikipedia pages and then aligning specific sentences, they use the explicit gendered pronouns in English (which are required by grammar) to "label" the dropped or neutral pronouns in the corresponding Spanish text.
- BERT Classification: They fine-tune a BERT model on this newly created dataset. The model is trained to look at a Spanish sentence and its surrounding context (up to 128 tokens) to predict whether a missing or neutral pronoun should be masculine or feminine.
- NMT Integration: Instead of redesigning the entire translation architecture, they use a "tagging" approach. They take a standard sentence-level Transformer model and simply append a gender tag (like
<FEM>or<MASC>) produced by their BERT classifier to the input.
Data and Evaluation
The researchers utilized Spanish and English Wikipedia for cross-lingual pivoting, yielding a dataset of approximately 79,000 pro-drop examples and 187,000 possessive examples. For evaluation, they used the WMT’13 (Spanish-English) benchmark and a human evaluation set of 993 examples to verify the quality of the automated "pivot" labels.
Key Results
- Classifier Accuracy: The fine-tuned BERT classifier achieved a 92% F1 score for Spanish dropped feminine pronouns. For comparison, standard NMT models only achieved 30-51%, and a non-fine-tuned BERT managed 54-71%.
- Translation Improvement: Adding the BERT-generated gender tags to a standard NMT model resulted in an 8.8% F1 improvement in the generation of feminine pronouns in the final English translation.
- Data Quality: Human evaluators agreed with the automatically generated "pivot" labels 98% of the time for pro-drop cases, proving the pivoting technique is remarkably high-quality.
Conclusion and Future Directions
The "pivoting" hack demonstrates the power of using the grammatical requirements of one language to supervise the learning of another. This allows for the creation of "silver-standard" data at a scale that would be impossible with human annotators. While the model performed excellently on Wikipedia-style text, future work could address domain shifts and explore zero-shot potential for low-resource languages where parallel data is scarce.
Summary updated