EAG: Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation

Yulin Xu, Zhen Yang, Fandong Meng, Jie Zhou

Introduction

Multilingual Neural Machine Translation (MNMT) aims to translate between multiple languages using a single model. However, most existing models suffer from a significant "English-centric" bias. Because the vast majority of parallel training data pairs English with another language (e.g., English-French), direct translation between non-English pairs (e.g., French-German) often relies on "pivoting" through English or zero-shot capabilities, which are frequently suboptimal.

To address this, researchers have explored two main avenues: extraction-based methods and generation-based methods. Extraction-based methods identify triplets where the same English sentence appears in two different bilingual pairs. However, identical sentences are rare, leading to data scarcity. Generation-based methods use models to synthesize data, but these often lack the linguistic diversity of natural language. Researchers from Tencent have introduced EAG (Extract and Generate), a novel two-step pipeline designed to scale up high-quality training data for non-English language pairs by combining the scalability of generation with the quality of real-world extraction.

The EAG Framework

The EAG approach utilizes a clever two-step pipeline to create multi-way aligned corpora from existing bilingual data.

The EAG pipeline consists of a loose extraction step followed by a neural refinement step to ensure perfect alignment.

Step 1: Extraction (The "Rough Match")

Instead of requiring identical English pivot sentences, EAG employs a relaxed constraint using Edit Distance. It identifies "nearly identical" English sentences across different bilingual corpora. For instance, as shown in Figure 1, "You want justice, right?" and "You want out, right?" are flagged as a candidate pair. This "looser" constraint allows researchers to find significantly more data than previous methods—up to 4x for certain language pairs like German-Russian.

Step 2: Generation (The "Refinement")

Because the extracted sentences are not perfectly identical, their corresponding translations in other languages are not perfectly aligned. EAG utilizes a specialized Generation Model ($m$) to bridge this gap. This model is trained via self-supervision to take a source sentence and a "noisy" version of a target sentence, refining the latter to match the source's meaning exactly. In the example above, the model modifies a Chinese translation of "You want out" to perfectly align with the English "You want justice."

Experimental Results

The effectiveness of EAG was evaluated on the WMT-5 benchmark (covering Czech, German, Spanish, French, and Russian) and the massive OPUS-100 dataset.

Performance comparison of EAG against various baselines across multiple language directions.

As shown in Figure 6, EAG consistently outperforms existing state-of-the-art methods. On the WMT-5 benchmark, it achieved an average improvement of +1.1 BLEU points over the previous best extraction-based method. On the OPUS-100 dataset, the improvement reached +1.4 BLEU points, demonstrating the method's scalability. Notably, in specific directions like Russian-to-German, EAG saw gains of up to 2.4 BLEU points, a substantial jump in translation quality.

Key Insights and Ideas

  • Diversity is King: Purely generated (synthetic) data often lacks the "flavor" and variety of human language. By starting with real human-translated sentences and only refining them, EAG preserves the structural diversity of the original bilingual corpora.
  • The Similarity-Noise Balance: The researchers found that the model performs best when the "noise" ratio in training ($\beta$) matches the similarity threshold used in extraction ($\gamma$). If the noise is too high, the model fails to converge; if it's too low, it cannot handle the differences between the extracted sentences.
  • Superiority over Pivoting: EAG-trained models consistently beat traditional "bridging" or pivot-based systems (which translate to English first, then to the target language), avoiding the compounded errors inherent in two-step translation.

Conclusion

EAG represents a significant step forward in mitigating the English-centric bias in MNMT. By intelligently combining data extraction and neural refinement, it provides a scalable way to generate high-quality, multi-way aligned corpora. While the pipeline involves higher computational costs than simple extraction, the substantial gains in translation quality justify the investment. Future directions include exploring EAG for domain adaptation and investigating different neural architectures for the generation model to further improve refinement quality.

Summary updated

Abstract

Complete Multi-lingual Neural Machine Translation (C-MNMT) achieves superior performance against the conventional MNMT by constructing multi-way aligned corpus, i.e., aligning bilingual training examples from different language pairs when either their source or target sides are identical. However, since exactly identical sentences from different language pairs are scarce, the power of the multi-way aligned corpus is limited by its scale. To handle this problem, this paper proposes "Extract and Generate" (EAG), a two-step approach to construct large-scale and high-quality multi-way aligned corpus from bilingual data. Specifically, we first extract candidate aligned examples by pairing the bilingual examples from different language pairs with highly similar source or target sentences; and then generate the final aligned examples from the candidates with a well-trained generation model. With this two-step pipeline, EAG can construct a large-scale and multi-way aligned corpus whose diversity is almost identical to the original bilingual corpus. Experiments on two publicly available datasets i.e., WMT-5 and OPUS-100, show that the proposed method achieves significant improvements over strong baselines, with +1.1 and +1.4 BLEU points improvements on the two datasets respectively.
BibTeX
@inproceedings{xu2022eag,
  author = {Yulin Xu and Zhen Yang and Fandong Meng and Jie Zhou},
  title = {{EAG: Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.560},
}
Figure