Self-Induced Curriculum Learning in Neural Machine Translation

Dana Ruiter, Cristina España-Bonet, Josef van GenabitharXiv 2020 · arXiv

TL;DR

This paper introduces Self-Induced Curriculum Learning (SSNMT) for neural machine translation, where the model automatically learns to select the most beneficial training data over time. This approach allows for effective training on comparable (rather than parallel) corpora, leading to translation quality on par with supervised methods without manual data curation.

What Problem Does This Paper Address?

Traditional machine translation (MT) systems, especially neural MT, heavily rely on vast amounts of parallel data (sentences and their direct translations). Acquiring such parallel data is expensive and time-consuming. While unsupervised MT (USNMT) has emerged to address this, it often requires massive monolingual datasets. This paper tackles the challenge of training effective MT systems using readily available comparable corpora (documents on the same topic but not direct translations) by developing a method where the model itself learns to identify and select the most useful sentence pairs for training, effectively creating its own learning curriculum.

Methods

The core of the proposed method is Self-Supervised Neural Machine Translation (SSNMT). SSNMT employs a bidirectional NMT system that simultaneously learns to translate from language L1 to L2 and vice-versa. This architecture encodes sentences into a shared semantic space. The system then uses the internal representations of this NMT model to score sentence pairs from comparable corpora. Pairs that are deemed similar enough by both translation directions are selected for training. This selection process is dynamic and happens online, meaning the model's ability to select good data improves as it learns to translate. This creates a "self-induced curriculum" where the model progressively selects more relevant and complex sentence pairs. The authors also analyze the characteristics of these self-selected data points, looking at complexity and relevance.

Datasets

  • Wikipedia (WP): Used as a comparable corpus for training and evaluating the SSNMT system. English, French, German, and Spanish dumps were utilized.
  • Europarl (EP): Used for control experiments and to create synthetic comparable corpora for evaluating data extraction quality.
  • WikiMatrix: Used for comparison with SSNMT, consisting of pre-selected mined sentence pairs.

Key Insights and Ideas

  • Automatic Curriculum Generation: SSNMT demonstrates that a neural network can inherently learn to create its own curriculum, selecting simpler examples first and gradually progressing to more complex ones, mimicking human learning.
  • Mutual Supervision: The interaction between the NMT model's internal representations for source and target languages is crucial for both high-precision data extraction and effective translation. This mutual supervision drives the self-induced curriculum.
  • Denoising Curriculum: The process of selecting increasingly accurate sentence pairs acts as a form of denoising curriculum, where the proportion of non-translation pairs decreases significantly during training.
  • Task-Relevance Evolution: The self-selected data evolves to become increasingly relevant to the translation task as training progresses, leading to improved translation quality.
  • Complexity Progression: The system naturally selects sentences of increasing complexity, starting with content suitable for high school students and moving towards undergraduate-level material.

Key Results

  • SSNMT achieves translation quality on par with current state-of-the-art supervised NMT systems trained on externally preselected mined parallel data (WikiMatrix).
  • For en-fr, SSNMT outperforms supervised NMT trained on all extracted data by 1-3 BLEU points, suggesting the system's ability to filter out false positives over time.
  • The data extraction process shows a decrease in non-matching pairs from 18% to 2% during training.
  • The complexity of extracted sentences, measured by the Gunning Fog Index, shows a clear progression from high school to undergraduate level.

Limitations and Future Work

  • Language Script Dependence: The authors raise questions about how SSNMT will perform on languages that do not share a script or when using smaller BPE sizes, suggesting this as a direction for future work.
  • Homograph Importance: The reliance on homographs for initialization in the early stages of training is noted, and its implications for languages with fewer homographs or different script systems warrant further investigation.
  • WikiMatrix Thresholding: The performance comparison with WikiMatrix highlights the challenge of finding optimal thresholds for mined sentence pairs, a problem SSNMT avoids by using its internal representations.

Summary updated

Abstract

Self-supervised neural machine translation (SSNMT) jointly learns to identify and select suitable training data from comparable (rather than parallel) corpora and to translate, in a way that the two tasks support each other in a virtuous circle. In this study, we provide an in-depth analysis of the sampling choices the SSNMT model makes during training. We show how, without it having been told to do so, the model self-selects samples of increasing (i) complexity and (ii) task-relevance in combination with (iii) performing a denoising curriculum. We observe that the dynamics of the mutual-supervision signals of both system internal representation types are vital for the extraction and translation performance. We show that in terms of the Gunning-Fog Readability index, SSNMT starts extracting and learning from Wikipedia data suitable for high school students and quickly moves towards content suitable for first year undergraduate students.
BibTeX
@misc{ruiter2020self,
  author = {Dana Ruiter and Cristina España-Bonet and Josef van Genabith},
  title = {{Self-Induced Curriculum Learning in Neural Machine Translation}},
  year = {2020},
  eprint = {2004.03151},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.03151},
}
Figure