Introduction
Multilingual Neural Machine Translation (MNMT) aims to translate between multiple languages using a single model. However, most existing models suffer from a significant "English-centric" bias. Because the vast majority of parallel training data pairs English with another language (e.g., English-French), direct translation between non-English pairs (e.g., French-German) often relies on "pivoting" through English or zero-shot capabilities, which are frequently suboptimal.
To address this, researchers have explored two main avenues: extraction-based methods and generation-based methods. Extraction-based methods identify triplets where the same English sentence appears in two different bilingual pairs. However, identical sentences are rare, leading to data scarcity. Generation-based methods use models to synthesize data, but these often lack the linguistic diversity of natural language. Researchers from Tencent have introduced EAG (Extract and Generate), a novel two-step pipeline designed to scale up high-quality training data for non-English language pairs by combining the scalability of generation with the quality of real-world extraction.
The EAG Framework
The EAG approach utilizes a clever two-step pipeline to create multi-way aligned corpora from existing bilingual data.
Step 1: Extraction (The "Rough Match")
Instead of requiring identical English pivot sentences, EAG employs a relaxed constraint using Edit Distance. It identifies "nearly identical" English sentences across different bilingual corpora. For instance, as shown in Figure 1, "You want justice, right?" and "You want out, right?" are flagged as a candidate pair. This "looser" constraint allows researchers to find significantly more data than previous methods—up to 4x for certain language pairs like German-Russian.
Step 2: Generation (The "Refinement")
Because the extracted sentences are not perfectly identical, their corresponding translations in other languages are not perfectly aligned. EAG utilizes a specialized Generation Model ($m$) to bridge this gap. This model is trained via self-supervision to take a source sentence and a "noisy" version of a target sentence, refining the latter to match the source's meaning exactly. In the example above, the model modifies a Chinese translation of "You want out" to perfectly align with the English "You want justice."
Experimental Results
The effectiveness of EAG was evaluated on the WMT-5 benchmark (covering Czech, German, Spanish, French, and Russian) and the massive OPUS-100 dataset.
As shown in Figure 6, EAG consistently outperforms existing state-of-the-art methods. On the WMT-5 benchmark, it achieved an average improvement of +1.1 BLEU points over the previous best extraction-based method. On the OPUS-100 dataset, the improvement reached +1.4 BLEU points, demonstrating the method's scalability. Notably, in specific directions like Russian-to-German, EAG saw gains of up to 2.4 BLEU points, a substantial jump in translation quality.
Key Insights and Ideas
- Diversity is King: Purely generated (synthetic) data often lacks the "flavor" and variety of human language. By starting with real human-translated sentences and only refining them, EAG preserves the structural diversity of the original bilingual corpora.
- The Similarity-Noise Balance: The researchers found that the model performs best when the "noise" ratio in training ($\beta$) matches the similarity threshold used in extraction ($\gamma$). If the noise is too high, the model fails to converge; if it's too low, it cannot handle the differences between the extracted sentences.
- Superiority over Pivoting: EAG-trained models consistently beat traditional "bridging" or pivot-based systems (which translate to English first, then to the target language), avoiding the compounded errors inherent in two-step translation.
Conclusion
EAG represents a significant step forward in mitigating the English-centric bias in MNMT. By intelligently combining data extraction and neural refinement, it provides a scalable way to generate high-quality, multi-way aligned corpora. While the pipeline involves higher computational costs than simple extraction, the substantial gains in translation quality justify the investment. Future directions include exploring EAG for domain adaptation and investigating different neural architectures for the generation model to further improve refinement quality.
Summary updated