Modern Neural Machine Translation (NMT) models are often viewed as "black boxes" that learn to map source sentences to target translations in an end-to-end fashion. While this approach has yielded state-of-the-art results, it obscures the internal mechanics of how these models acquire translation competence. In this work, Elena Voita and colleagues (EMNLP 2021) perform a "post-mortem" of the NMT training process, interpreting neural model development through the components of classical Statistical Machine Translation (SMT): language modeling, lexical translation, and reordering.
The NMT Training Curriculum
The authors demonstrate that NMT training is not a uniform process but rather follows a predictable curriculum of three distinct stages. This progression is consistent across different architectures (Transformer, LSTM) and modeling paradigms.
Methodology: Two Viewpoints
To identify these stages, the researchers analyzed training from two perspectives:
- The Abstract Viewpoint: Using Layerwise Relevance Propagation (LRP), they quantified how much the model relies on the source sentence versus the target prefix (previously generated words) at each step.
- The Practical Viewpoint: They evaluated checkpoints using SMT-inspired metrics, including KenLM scores for fluency, token-level accuracy for lexical mapping, and Kendall Tau for reordering complexity.
The Three Stages of Competence
Stage 1: The Target Language Model Phase
In the earliest phase of training, the model focuses almost exclusively on the target language. It "hallucinates" output by generating frequent n-grams and common phrases that sound natural in the target language but bear little to no relation to the source input. During this stage, entropy of source contributions is high because the model has not yet learned to attend to specific source tokens.
Stage 2: The Lexical Translation Phase
As training progresses, the model begins to incorporate source information. This stage is characterized by a sharp increase in translation quality (BLEU) and lexical accuracy. However, the translations are largely "monotonic"—the model translates word-for-word, strictly following the word order of the source sentence. The model has learned what words to use but not yet where to place them in complex structures.
Stage 3: The Reordering Phase
In the final stage, standard metrics like BLEU often plateau, suggesting that learning has slowed. However, the authors find that the model is still undergoing significant qualitative shifts: it is mastering complex reordering patterns. The model moves away from monotonic translation to handle the nuanced syntactic requirements of the target language, such as verb-final structures in German.
Applications: Improving Non-Autoregressive Translation
This discovery has immediate practical implications for Non-Autoregressive Translation (NAT). NAT models often struggle with "multi-modality"—the challenge of choosing between multiple valid word orders. By using a "simpler" teacher (a checkpoint from the end of Stage 2 that is more monotonic) for knowledge distillation, the authors were able to provide a more consistent signal to the NAT student. This strategy improved a vanilla NAT model's performance by over 1.1 BLEU points compared to using a fully converged, complex teacher.
Conclusion
By decomposing the NMT training process into SMT-like competencies, this research reveals that neural models naturally prioritize fluency, then adequacy, and finally structural refinement. Understanding this internal curriculum not only provides theoretical insight into deep learning but also offers a new lever for optimizing complex translation architectures.
Summary updated