Introduction
Neural Machine Translation (NMT) has achieved state-of-the-art performance across nearly all language pairs, yet it remains a "black box." Unlike older Statistical Machine Translation (SMT) models that relied on explicit, human-readable phrase tables (dictionaries of word and phrase mappings), NMT models store their knowledge in millions of abstract weights. This lack of transparency complicates error analysis: when a model fails, it is often unclear whether the failure stems from a lack of vocabulary knowledge or an inability to handle complex grammar. This research bridges the gap by extracting interpretable phrase tables from NMT models to assess the specific bilingual knowledge they internalize during training.
Methodology: Extracting the Neural Phrase Table
The researchers developed a novel extraction method inspired by the concept of "forgetting" in image classification. The process involves several key steps:
- Force-Decoding: A trained NMT model is tasked with predicting the target sentences of its own training data.
- Masking: If the model correctly predicts a word in a sentence, that word is marked as "learned." If it fails, the word is masked out.
- Phrase Extraction: A standard SMT pipeline (using the Moses toolkit) is run on this masked data. Only phrase pairs that the NMT model successfully "memorized" are extracted into a phrase table.
- Validation: To ensure the extracted table accurately represents the NMT model's internal state, the authors build a mini-SMT model using only these phrases. A high correlation between the SMT model's performance and the NMT model's performance over time validates the extraction.
Experimental Setup and Datasets
The study evaluates this method across diverse language pairs to ensure generalizability:
- WMT2014 English-German (En-De): A standard large-scale benchmark for translation.
- WAT2017 English-Japanese (En-Ja): Used to test the method on syntactically distant languages with significantly different word orders.
- IWSLT14 English-German: A smaller dataset used for domain adaptation experiments, such as moving from News to Speech domains.
Key Insights: NMT as a Knowledge Distiller
The extraction process revealed several surprising characteristics of how NMT models learn:
The Simple-to-Complex Curriculum
NMT models naturally follow a curriculum, learning easy patterns first. In early training epochs, they master short phrases, 1-1 word alignments, and monotone (word-for-word) translations. Only later do they tackle complex reordering and long-tail vocabulary.
Data Cleaning and Distillation
NMT models do not just memorize training data; they act as filters. While a raw dataset might contain 17.5M phrases, an NMT model might only learn 9.0M of them. Crucially, these "distilled" phrases are of significantly higher quality. In the En-De task, the NMT-distilled table achieved nearly identical translation quality (17.90 vs 17.91 BLEU) despite having 50% fewer phrases.
Capacity vs. Knowledge
Increasing model size (e.g., from Transformer-Base to Transformer-Big) does not significantly increase the "vocabulary" of phrases learned. Instead, larger models appear to improve their "composition rules"—the ability to glue phrases together correctly—rather than simply memorizing more mappings.
Results and Learning Dynamics
The quality of the extracted phrase tables showed a Pearson correlation of over 0.95 with actual NMT BLEU scores, proving that the phrase table is a valid proxy for the model's internal knowledge.
Furthermore, the study observed "catastrophic forgetting": approximately 20% of the bilingual phrases a model learns during early training are predicted incorrectly by the time the model reaches convergence. Techniques like Back-Translation (BT) were shown to mitigate this by both introducing massive amounts of new phrase knowledge and stabilizing existing mappings.
Conclusion
By extracting interpretable phrase tables, this research provides a window into the NMT learning process. While phrase tables primarily capture lexical knowledge, they offer a powerful tool for error diagnosis and understanding the "distillation" effect of neural models. Future work will look toward extracting more complex structures, such as syntax rules, to provide a more complete picture of the neural translation engine.
Summary updated