Abstract
Self-Supervised Neural Machine Translation (SSNMT) models training on comparable corpora naturally induce a learning curriculum without human guidance. This paper analyzes how these models evolve from simple word-matching to complex sentence translation, effectively performing self-directed denoising and complexity-based scheduling. We demonstrate that the model naturally constructs a curriculum of increasing difficulty and relevance, mimicking human learning patterns.
Introduction
Standard Neural Machine Translation (NMT) is notoriously data-hungry, typically requiring massive parallel corpora. While unsupervised NMT exists, it often demands enormous monolingual datasets. SSNMT offers a middle ground by utilizing "comparable" corpora—such as Wikipedia articles in different languages on the same topic. These contain "hidden" parallel sentences. The core challenge is the simultaneous task of identifying these parallel sentences and learning to translate them. This study investigates the emergent curriculum within this "black box" process.
Methods and Framework
The authors analyze an SSNMT framework based on a bidirectional Transformer. The system employs two internal representation types to score sentence pairs:
- $C_w$ (Word-based): Static word embeddings.
- $C_h$ (Context-based): Dynamic encoder outputs that evolve during training.
A "virtuous circle" emerges: as translation improves, internal representations become more accurate, allowing the model to select higher-quality training data, which further refines translation capability.
The Self-Induced Curriculum
The research identifies three primary types of self-induced curricula:
1. Denoising and Task-Relevance
The model automatically reduces "noise" (non-parallel pairs) over time. Initially, the model's ability to distinguish parallel sentences is limited, but it rapidly improves as the encoder learns better representations.
As shown in Figure 1, the extraction precision reaches 0.96–0.99 on synthetic tests, indicating the model becomes nearly perfect at identifying true translations within noisy data.
2. The Homograph Bootstrap
In the initial stages, the model relies on homographs—words spelled identically across languages (e.g., "Paris"). These serve as anchors to initiate the learning process when the model is otherwise "clueless."
3. Linguistic Complexity
By measuring the Gunning Fog Index, the authors found that the model starts with text suitable for high school levels and progresses to undergraduate-level complexity.
Figure 6 illustrates this progression: the top panel shows the increasing readability index, while the bottom panel shows the decreasing percentage of homographs in selected batches as the model moves beyond simple word-matching to more complex linguistic structures.
Key Results
- Performance: SSNMT achieves BLEU scores (e.g., ~29.5 for En-Fr) comparable to state-of-the-art unsupervised systems while using significantly less data.
- Mutual Supervision: $C_w$ guides early learning when the encoder ($C_h$) is untrained. Later, $C_h$ takes over to identify nuanced, non-literal translations.
- Curriculum Advantage: Models trained on this self-induced curriculum often outperform supervised models trained on the entire aggregate dataset, suggesting that filtering out early noisy samples is a feature, not a bug.
Limitations and Future Work
The primary limitation is the "script barrier"; languages with different scripts (e.g., English and Chinese) lack the homograph anchors needed for bootstrapping. Additionally, the complexity curriculum is bounded by the source corpus (Wikipedia). Future work may explore BPE-based anchors or methods for distant language pairs.
Summary updated