EAG: Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation

Yulin Xu, Zhen Yang, Fandong Meng, Jie Zhou

Introduction

Multilingual Neural Machine Translation (MNMT) has emerged as a powerful paradigm for translating between dozens or even hundreds of languages within a single model. However, most existing MNMT systems suffer from a significant "English-centric" bias. Because the majority of available training data consists of English paired with another language (e.g., English-French), models often lack direct supervision for non-English pairs (e.g., French-German). This results in poor performance for "zero-shot" or direct non-English translation, often requiring inefficient "pivoting" through English at inference time.

To address this, researchers have explored Complete MNMT (C-MNMT), which seeks to create multi-way aligned corpora. Traditional C-MNMT relies on finding instances where the English side of two different language pairs is exactly identical. While precise, this requirement is overly restrictive, leading to extremely small datasets that fail to capture the linguistic diversity of real-world text. This paper introduces the Extract and Generate (EAG) framework, a novel approach that bridges the gap between data quantity and alignment quality.

The EAG Framework

The EAG framework operates on the principle that "roughly aligned" data is a valuable starting point that can be refined into high-quality training material. The process consists of two primary phases: Extraction and Generation.

1. Extract: The Rough Alignment Phase

Instead of requiring exact string identity between English sentences in different pairs, EAG utilizes a similarity-based extraction method. By using Edit Distance as a metric, the framework identifies English sentences that are structurally and semantically similar, even if they are not identical. This loosening of constraints significantly expands the pool of candidate aligned examples, increasing the available training data for specific language pairs by as much as 400% compared to exact-match methods.

2. Generate: The Refinement Phase

Because the extracted sentences are only similar and not identical, the corresponding non-English translations are not perfectly aligned. To correct this, EAG employs a specialized refinement model ($m$). This model is trained using a self-supervised "noising" strategy: it learns to take a source sentence and a "noisy" (roughly aligned) target sentence, then refines the target to be a perfect semantic match for the source.

By applying this refinement model to the candidates from the extraction phase, EAG produces a large-scale, multi-way aligned corpus. This corpus retains the lexical and structural diversity of the original human-written text while ensuring the precise alignment necessary for high-quality NMT training.

Experimental Setup and Results

The researchers evaluated the EAG framework on two major benchmarks: WMT-5 (covering Czech, German, Spanish, French, Russian, and English) and OPUS-100 (a massive corpus covering 100 languages).

Performance Gains

EAG demonstrated consistent improvements across both benchmarks:

  • WMT-5: The framework achieved an average improvement of +1.1 BLEU points over the standard extraction-based C-MNMT baseline.
  • OPUS-100: The method showed strong scalability, outperforming the baseline by +1.4 BLEU points.

Data Efficiency and Zero-Shot Translation

One of the most striking findings was EAG's data efficiency. The framework achieved results comparable to state-of-the-art models trained on significantly larger, proprietary datasets, despite using 75 times less data. Furthermore, by providing direct (synthetic) training data for non-English pairs, EAG significantly outperformed traditional pivoting methods, which are the current industry standard for low-resource translation pairs.

Discussion and Insights

The success of EAG highlights a critical synergy between the similarity threshold ($\gamma$) used during extraction and the noise ratio ($\beta$) used to train the refinement model. If extraction is too loose, the refinement model requires greater denoising capabilities to maintain alignment quality.

Unlike standard back-translation, which can produce repetitive or "boring" synthetic text, EAG preserves the diversity of the original human-written sentences. The refinement step acts surgically, making only the necessary changes to ensure alignment without stripping away the natural variety of the source data.

Conclusion

The EAG framework provides a scalable and effective solution to the English-centric bottleneck in multilingual translation. By moving beyond the constraints of exact-match pivoting and leveraging neural refinement, EAG enables the creation of high-quality, multi-way aligned corpora from existing bilingual data. Future work will explore the application of EAG to domain adaptation and the development of more sophisticated neural architectures for the refinement process.

Summary updated

BibTeX
@inproceedings{xu2022eag,
  author = {Yulin Xu and Zhen Yang and Fandong Meng and Jie Zhou},
  title = {{EAG: Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.560},
}
Figure