Abstract
Synthetic translations have been used for a wide range of NLP tasks primarily as a means of data augmentation. This work explores, instead, how synthetic translations can be used to revise potentially imperfect reference translations in mined bitext. We find that synthetic samples can improve bitext quality without any additional bilingual supervision when they replace the originals based on a semantic equivalence classifier that helps mitigate NMT noise. The improved quality of the revised bitext is confirmed intrinsically via human evaluation and extrinsically through bilingual induction and MT tasks.
Introduction
Large-scale parallel corpora, or "bitext," are the foundation of modern machine translation (MT). However, these datasets are typically mined from the web using automated tools that identify related sentences across languages. While effective at scale, these tools often capture "comparable" rather than strictly "parallel" pairs—sentences that share a topic but differ in specific details, such as names, numbers, or additional clauses.
Traditionally, researchers have addressed this noise through data filtering (removing low-quality pairs) or data augmentation (adding synthetic machine-translated data to increase volume). This paper proposes a third path: data revision. By using Neural Machine Translation (NMT) models to generate synthetic alternatives and a semantic scorer to decide when to replace the original human-mined text, the authors demonstrate that synthetic data can actually "repair" a noisy corpus.
Methodology: The Self-Correction Pipeline
The authors develop a pipeline that improves bitext quality without requiring any additional human supervision or gold-standard reference data. The process follows four primary steps:
- Initial Model Training: Two standard NMT models are trained on the original, noisy mined bitext: a forward model ($S \to T$) and a backward model ($T \to S$).
- Candidate Generation: For every sentence pair in the original corpus, the models generate two synthetic candidates: a forward translation of the source and a backward translation of the target.
- Semantic Scoring: The original pair and the synthetic candidates are evaluated using divergentmBERT, a pre-trained model specifically designed to detect semantic divergences between cross-lingual sentence pairs.
- Selective Replacement: The system compares the semantic equivalence scores. If a synthetic candidate achieves a higher score than the original human-mined pair (by a predefined margin), the original is replaced. Otherwise, the original is retained.
This approach leverages the "generalization" property of NMT models. While the training data is noisy, the model learns the most frequent and consistent patterns across the entire corpus, effectively "voting out" the idiosyncratic errors found in individual mined pairs.
Experimental Setup
The study evaluated the revision strategy on two language pairs from the WikiMatrix corpus:
- English-Greek (EN-EL): A linguistically distant pair with approximately 750,000 pairs.
- English-Romanian (EN-RO): A more similar pair with approximately 582,000 pairs.
The quality of the revised bitext was assessed through human evaluation, downstream MT performance (using the TED corpus), and Bilingual Lexicon Induction (BLI) via the MUSE dataset.
Results and Insights
Human Evaluation
In blind tests, bilingual speakers compared the original WikiMatrix references against the synthetic candidates. The synthetic versions were judged superior 60% of the time. When the semantic scorer was highly confident (a margin $> 5$), the preference for synthetic translations rose to 87.5%. This confirms that NMT models can indeed produce more accurate translations than the "human" data they were trained on when that data is mined rather than hand-translated.
Downstream Performance
The revision process yielded significant gains across multiple tasks:
- Machine Translation: Replacing noisy data with synthetic revisions led to a +1.49 BLEU point improvement for English-Greek.
- Bilingual Lexicon Induction: The revised data improved both precision and recall in dictionary induction, particularly for low- and medium-frequency words which are typically difficult to learn from noisy data.
- Word Alignment: The authors observed a 6-7% increase in alignment coverage, suggesting that synthetic revisions create a more consistent mapping between languages, making it easier for models to learn word-level correspondences.
The Role of Bi-directionality
A key finding was that forward and backward translations often correct different types of errors. By using both directions and selecting the best candidate via the semantic scorer, the system achieved better results than using a single direction alone.
Discussion and Limitations
While the revision strategy is effective, it introduces specific challenges:
- Gender Bias: NMT models often default to masculine pronouns or stereotypical gender roles. The authors found that replacing human data with synthetic data slightly increased these biases in the final dataset.
- Resource Requirements: The method assumes the initial noisy data is sufficient to train a "decent" NMT model. In extremely low-resource scenarios, the synthetic candidates might be too poor to offer an improvement.
- Computational Overhead: Generating and scoring candidates for an entire corpus is computationally expensive compared to simple filtering.
Conclusion
This work shifts the perspective on synthetic data from a tool for expansion to a tool for refinement. By treating NMT models as "denoisers" and using semantic scorers to gate the revision process, researchers can significantly improve the quality of web-mined bitext. This "self-correction" loop offers a scalable way to enhance datasets without the need for additional human labor.
Summary updated