Abstract: Despite advances in neural machine translation (NMT) quality, rare words continue to be problematic. For humans, the solution to the rare-word problem has long been dictionaries, but dictionaries cannot be straightforwardly incorporated into NMT. In this paper, we describe a new method for "attaching" dictionary definitions to rare words so that the network can learn the best way to use them. We demonstrate improvements of up to 1.8 BLEU using bilingual dictionaries.
Note: This paper was retracted following its publication at WMT 2020 due to issues identified in the evaluation process. The following summary describes the methods and claims as presented in the original manuscript.
Introduction
Neural Machine Translation (NMT) systems frequently struggle with rare words. Due to Zipf’s Law, a significant portion of language consists of low-frequency vocabulary. While subword units like Byte Pair Encoding (BPE) mitigate the "unknown token" problem, they often fail to provide sufficient semantic context for the model to translate rare concepts accurately. Humans typically resolve this by consulting a dictionary. However, integrating dictionary entries into NMT is non-trivial; simply appending definitions to training data often introduces noise that degrades performance.
The "Attach" Method
To address this, the authors proposed the Attach method, which extends the Transformer architecture to dynamically consult dictionary definitions.
The core innovation lies in Nonlinear Position Encodings. In a standard Transformer, tokens follow a linear sequence. The Attach method uses a "tree-like" structure:
- Definition Retrieval: For a rare word at position $p$, the system retrieves its definition (up to 50 tokens).
- Encoding: A definition word $d$ at position $q$ is encoded by combining the embedding of the rare word, its position $p$, the embedding of $d$, and its relative position $q$ within the definition.
- Dynamic Learning: This allows the self-attention mechanism to treat the definition as an extension of the rare word. The model learns to extract relevant information while ignoring meta-linguistic noise common in dictionaries.
Experimental Results
The method was evaluated on Chinese-English (UM-Corpus) and German-English (Europarl V7) tasks.
In the Chinese-English Spoken domain, the researchers reported gains of up to +3.1 BLEU. The model demonstrated a remarkable ability for morphological adaptation, such as correctly inflecting a verb defined in the infinitive to match the sentence's tense. Furthermore, the use of monolingual dictionaries (source-language definitions) also yielded improvements, suggesting that extra semantic context helps the encoder build more robust representations of rare words.
Limitations and Retraction
Despite the promising architectural concept, the method faces practical challenges. The inclusion of definitions significantly increases sequence length, making training and inference 2 to 3 times slower. Additionally, the system currently relies on exact headword matching, which may miss inflected forms.
Crucially, this paper was retracted because the reported BLEU scores were found to be unreliable due to errors in the evaluation pipeline. While the quantitative results should be viewed with caution, the idea of using nonlinear position encodings to attach external metadata remains a noteworthy contribution to NMT research.
Summary updated