Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT

Elena Voita, Rico Sennrich, Ivan Titov

Abstract

Differently from the traditional statistical MT that decomposes the translation task into distinct separately learned components, neural machine translation uses a single neural network to model the entire translation process. Despite neural machine translation being de-facto standard, it is still not clear how NMT models acquire different competences over the course of training, and how this mirrors the different models in traditional SMT. In this work, we look at the competences related to three core SMT components and find that during training, NMT first focuses on learning target-side language modeling, then improves translation quality approaching word-by-word translation, and finally learns more complicated reordering patterns. We show that this behavior holds for several models and language pairs. Additionally, we explain how such an understanding of the training process can be useful in practice and, as an example, show how it can be used to improve vanilla non-autoregressive neural machine translation by guiding teacher model selection.

Introduction

Modern Neural Machine Translation (NMT) operates as an end-to-end "black box," handling fluency, adequacy, and syntax within a single architecture. This contrasts sharply with classical Statistical Machine Translation (SMT), which relied on interpretable, separate components: a language model for fluency, a translation model for lexical correspondence, and a reordering model for syntax. While NMT is the current gold standard, its internal training dynamics—specifically when and how it acquires these distinct competences—remain poorly understood. This research aims to demystify NMT training by identifying specific stages of competence acquisition that mirror classical SMT components.

Methodology

The study analyzes NMT training through two lenses:

  1. Internal Attribution: Using Layerwise Relevance Propagation (LRP) to measure the model's reliance on the source sentence versus the target prefix at various training steps.
  2. Output Analysis: Evaluating translations at different checkpoints using KenLM scores (fluency), BLEU/Token Accuracy (general quality), and Fuzzy Reordering/Kendall Tau (structural alignment).

Experiments were conducted on standard benchmarks, including WMT14 English-German (~5.8M pairs) and WMT English-Russian (~2.5M pairs), across Transformer and LSTM architectures.

The Three-Stage Curriculum

The research identifies a consistent three-stage trajectory in NMT training:

  • Stage 1: Fluency First. Initially, the model acts primarily as a language model. It ignores the source input and "hallucinates" frequent target-language phrases.
  • Stage 2: Word-by-Word Adequacy. The model begins attending to the source, rapidly improving translation quality by adopting a simple, monotonic word-for-word mapping.
  • Stage 3: Structural Refinement. In the final phase, the model masters complex reordering patterns to align with the natural syntax of the target language, even after standard metrics like BLEU have plateaued.
Evolution of model translations from early hallucination to structured output.

Key Results

Quantitative analysis confirms these stages. Language modeling scores peak very early in training, typically within the first 5,000 steps, before the model shifts focus to the source text.

LM scores over training batches, showing the early peak in target-side fluency (Stage 1).

The "rapid gain" phase (Stage 2) generally continues until approximately 40,000 steps. Beyond this point, the model enters a long "refinement" phase (Stage 3). Interestingly, while all architectures follow this curriculum, LSTMs and Decoder-only Transformers tend to produce more monotonic translations than standard Encoder-Decoder Transformers.

Practical Application: NAT Distillation

Understanding these stages has immediate practical benefits. Non-Autoregressive Translation (NAT) models often struggle with the complex reordering learned by fully trained NMT teachers. By selecting a teacher from the end of Stage 2 (approx. 40k steps)—which provides a simpler, more monotonic signal—the authors improved vanilla NAT performance by 1.1 BLEU points compared to using a fully converged teacher.

Conclusion

NMT training naturally decomposes into functional stages that mirror the engineered components of SMT. However, standard metrics like BLEU are often blind to the structural nuances learned in the final stage of training. Future work should focus on developing evaluation tools sensitive to structural adequacy and exploring how these training dynamics manifest in larger-scale models like LLMs.

Summary updated

Abstract

Differently from the traditional statistical MT that decomposes the translation task into distinct separately learned components, neural machine translation uses a single neural network to model the entire translation process. Despite neural machine translation being de-facto standard, it is still not clear how NMT models acquire different competences over the course of training, and how this mirrors the different models in traditional SMT. In this work, we look at the competences related to three core SMT components and find that during training, NMT first focuses on learning target-side language modeling, then improves translation quality approaching word-by-word translation, and finally learns more complicated reordering patterns. We show that this behavior holds for several models and language pairs. Additionally, we explain how such an understanding of the training process can be useful in practice and, as an example, show how it can be used to improve vanilla non-autoregressive neural machine translation by guiding teacher model selection.
BibTeX
@inproceedings{voita2021language,
  author = {Elena Voita and Rico Sennrich and Ivan Titov},
  title = {{Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=8uOvhjZZxZ},
}

Loading related content...

Figure