Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation

Dana Ruiter, Josef van Genabith, Cristina España-Bonet

Abstract

Self-Supervised Neural Machine Translation (SSNMT) models training on comparable corpora naturally induce a learning curriculum without human guidance. This paper analyzes how these models evolve from simple word-matching to complex sentence translation, effectively performing self-directed denoising and complexity-based scheduling. We demonstrate that the model naturally constructs a curriculum of increasing difficulty and relevance, mimicking human learning patterns.

Introduction

Standard Neural Machine Translation (NMT) is notoriously data-hungry, typically requiring massive parallel corpora. While unsupervised NMT exists, it often demands enormous monolingual datasets. SSNMT offers a middle ground by utilizing "comparable" corpora—such as Wikipedia articles in different languages on the same topic. These contain "hidden" parallel sentences. The core challenge is the simultaneous task of identifying these parallel sentences and learning to translate them. This study investigates the emergent curriculum within this "black box" process.

Methods and Framework

The authors analyze an SSNMT framework based on a bidirectional Transformer. The system employs two internal representation types to score sentence pairs:

  1. $C_w$ (Word-based): Static word embeddings.
  2. $C_h$ (Context-based): Dynamic encoder outputs that evolve during training.

A "virtuous circle" emerges: as translation improves, internal representations become more accurate, allowing the model to select higher-quality training data, which further refines translation capability.

The Self-Induced Curriculum

The research identifies three primary types of self-induced curricula:

1. Denoising and Task-Relevance

The model automatically reduces "noise" (non-parallel pairs) over time. Initially, the model's ability to distinguish parallel sentences is limited, but it rapidly improves as the encoder learns better representations.

Precision and recall of sentence pair extraction over training steps.

As shown in Figure 1, the extraction precision reaches 0.96–0.99 on synthetic tests, indicating the model becomes nearly perfect at identifying true translations within noisy data.

2. The Homograph Bootstrap

In the initial stages, the model relies on homographs—words spelled identically across languages (e.g., "Paris"). These serve as anchors to initiate the learning process when the model is otherwise "clueless."

3. Linguistic Complexity

By measuring the Gunning Fog Index, the authors found that the model starts with text suitable for high school levels and progresses to undergraduate-level complexity.

Evolution of linguistic complexity (Gunning Fog Index) and reliance on homographs during training.

Figure 6 illustrates this progression: the top panel shows the increasing readability index, while the bottom panel shows the decreasing percentage of homographs in selected batches as the model moves beyond simple word-matching to more complex linguistic structures.

Key Results

  • Performance: SSNMT achieves BLEU scores (e.g., ~29.5 for En-Fr) comparable to state-of-the-art unsupervised systems while using significantly less data.
  • Mutual Supervision: $C_w$ guides early learning when the encoder ($C_h$) is untrained. Later, $C_h$ takes over to identify nuanced, non-literal translations.
  • Curriculum Advantage: Models trained on this self-induced curriculum often outperform supervised models trained on the entire aggregate dataset, suggesting that filtering out early noisy samples is a feature, not a bug.

Limitations and Future Work

The primary limitation is the "script barrier"; languages with different scripts (e.g., English and Chinese) lack the homograph anchors needed for bootstrapping. Additionally, the complexity curriculum is bounded by the source corpus (Wikipedia). Future work may explore BPE-based anchors or methods for distant language pairs.

Summary updated

BibTeX
@inproceedings{ruiter2020self,
  author = {Dana Ruiter and Josef van Genabith and Cristina España-Bonet},
  title = {{Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.202},
}
Figure