Abstract
Despite advances in neural machine translation (NMT) quality, rare words continue to be problematic. For humans, the solution to the rare-word problem has long been dictionaries, but dictionaries cannot be straightforwardly incorporated into NMT. In this paper, we describe a new method for "attaching" dictionary definitions to rare words so that the network can learn the best way to use them. We demonstrate improvements of up to 1.8 BLEU using bilingual dictionaries.
Introduction
Despite the massive success of Transformer-based models, NMT still struggles with the "long tail" of language. Due to Zipf’s Law, rare words are paradoxically common in any given text, yet they often fall outside a model's fixed vocabulary (typically 10k–100k words). While techniques like Byte Pair Encoding (BPE) break rare words into smaller subwords to avoid unknown tokens, models still frequently fail to translate them correctly—a phenomenon often attributed to "catastrophic forgetting."
Historically, researchers added dictionaries to training data as extra parallel sentences. However, this is less effective for NMT because dictionary definitions are often descriptive or meta-linguistic rather than direct translations. This work proposes a method to let the model use a dictionary more like a human does: by looking up a specific word in context.
Methods
The proposed method builds upon the standard Transformer architecture by "attaching" definitions directly to the source sentence through a non-linear structure.
The "Attach" Mechanism
For a rare word at position $p$ in a sentence, the model attaches the first 50 tokens of its dictionary definition. To help the model distinguish between the sentence and the definition, a new encoding scheme is used:
- Standard Word: $E = \text{Word Embedding} + \text{Position Encoding}(p)$
- Definition Word: $E = \text{Source Word Embedding} + \text{Source Position}(p) + \text{Def Word Embedding} + \text{Def Position Encoding}(q)$
This approach essentially "pins" the definition to the specific source word it describes, allowing the encoder to process the definition as an attribute of the rare word.
Implementation Details
To manage the lookup process, the system uses a frequency threshold; words appearing fewer than $X$ times in the training set trigger a dictionary lookup. To handle inflected forms (e.g., "running" vs "run"), the authors employ normalized Levenshtein distance for fuzzy matching against dictionary headwords, optimized via Locality Sensitive Hashing (LSH).
Results and Analysis
The researchers evaluated the approach on Chinese-English (UM-Corpus) and German-English (Europarl V7) datasets across multiple domains.
Performance Gains
The "Attach" method consistently outperformed both the standard Transformer baseline and an "Append" baseline (where dictionary entries are simply added as extra training data). On word-based Chinese-English tasks, the method showed improvements of up to 1.8 BLEU.
Rare Word Fluency
Beyond BLEU scores, the authors used MacroF1 to track how well the model translates unique word types. Significant gains in MacroF1 indicate that the model is genuinely improving on the "long tail" of the vocabulary rather than just common words. Qualitative analysis further revealed that the model learns to adapt definitions to context, such as correctly inflecting a verb from a dictionary definition to match the sentence's tense.
Conclusion
This research demonstrates that providing NMT models with structured access to bilingual dictionaries significantly improves the translation of rare words. While the "Attach" method increases computational overhead—with training and decoding taking 2–3 times longer—the gains in translation quality and rare-word fluency are substantial. Future work may explore the integration of other external resources like thesauruses and gazetteers to further enhance model performance.
Summary updated