Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT

Elena Voita, Rico Sennrich, Ivan Titov

Modern Neural Machine Translation (NMT) models are often viewed as "black boxes" that learn to map source sentences to target translations in an end-to-end fashion. While this approach has yielded state-of-the-art results, it obscures the internal mechanics of how these models acquire translation competence. In this work, Elena Voita and colleagues (EMNLP 2021) perform a "post-mortem" of the NMT training process, interpreting neural model development through the components of classical Statistical Machine Translation (SMT): language modeling, lexical translation, and reordering.

The NMT Training Curriculum

The authors demonstrate that NMT training is not a uniform process but rather follows a predictable curriculum of three distinct stages. This progression is consistent across different architectures (Transformer, LSTM) and modeling paradigms.

Methodology: Two Viewpoints

To identify these stages, the researchers analyzed training from two perspectives:

  1. The Abstract Viewpoint: Using Layerwise Relevance Propagation (LRP), they quantified how much the model relies on the source sentence versus the target prefix (previously generated words) at each step.
  2. The Practical Viewpoint: They evaluated checkpoints using SMT-inspired metrics, including KenLM scores for fluency, token-level accuracy for lexical mapping, and Kendall Tau for reordering complexity.
Evolution of source contributions and entropy during training, highlighting the three distinct stages defined by shifts in information reliance.

The Three Stages of Competence

Stage 1: The Target Language Model Phase

In the earliest phase of training, the model focuses almost exclusively on the target language. It "hallucinates" output by generating frequent n-grams and common phrases that sound natural in the target language but bear little to no relation to the source input. During this stage, entropy of source contributions is high because the model has not yet learned to attend to specific source tokens.

Stage 2: The Lexical Translation Phase

As training progresses, the model begins to incorporate source information. This stage is characterized by a sharp increase in translation quality (BLEU) and lexical accuracy. However, the translations are largely "monotonic"—the model translates word-for-word, strictly following the word order of the source sentence. The model has learned what words to use but not yet where to place them in complex structures.

Qualitative evolution of model translations from early hallucination (Stage 1) to the emergence of lexical mapping (Stage 2).

Stage 3: The Reordering Phase

In the final stage, standard metrics like BLEU often plateau, suggesting that learning has slowed. However, the authors find that the model is still undergoing significant qualitative shifts: it is mastering complex reordering patterns. The model moves away from monotonic translation to handle the nuanced syntactic requirements of the target language, such as verb-final structures in German.

Applications: Improving Non-Autoregressive Translation

This discovery has immediate practical implications for Non-Autoregressive Translation (NAT). NAT models often struggle with "multi-modality"—the challenge of choosing between multiple valid word orders. By using a "simpler" teacher (a checkpoint from the end of Stage 2 that is more monotonic) for knowledge distillation, the authors were able to provide a more consistent signal to the NAT student. This strategy improved a vanilla NAT model's performance by over 1.1 BLEU points compared to using a fully converged, complex teacher.

Conclusion

By decomposing the NMT training process into SMT-like competencies, this research reveals that neural models naturally prioritize fluency, then adequacy, and finally structural refinement. Understanding this internal curriculum not only provides theoretical insight into deep learning but also offers a new lever for optimizing complex translation architectures.

Summary updated

Abstract

Differently from the traditional statistical MT that decomposes the translation task into distinct separately learned components, neural machine translation uses a single neural network to model the entire translation process. Despite neural machine translation being de-facto standard, it is still not clear how NMT models acquire different competences over the course of training, and how this mirrors the different models in traditional SMT. In this work, we look at the competences related to three core SMT components and find that during training, NMT first focuses on learning target-side language modeling, then improves translation quality approaching word-by-word translation, and finally learns more complicated reordering patterns. We show that this behavior holds for several models and language pairs. Additionally, we explain how such an understanding of the training process can be useful in practice and, as an example, show how it can be used to improve vanilla non-autoregressive neural machine translation by guiding teacher model selection.
BibTeX
@inproceedings{voita2021language,
  author = {Elena Voita and Rico Sennrich and Ivan Titov},
  title = {{Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=8uOvhjZZxZ},
}
Figure