Abstract
While external information like dictionaries or suggestions from other translation systems can boost Neural Machine Translation (NMT) performance, this data is often riddled with "noise" that can mislead the model. This paper introduces a dual-discriminator framework that identifies and filters out irrelevant external words at both the sentence level (global noise) and the specific word-generation step (local noise), making NMT significantly more robust to low-quality external inputs.
Introduction
Neural Machine Translation has become the industry standard, but it often struggles with rare words or domain-specific terminology. To address this, researchers frequently "inject" external information—such as bilingual dictionaries, outputs from older Statistical Machine Translation (SMT) systems, or human suggestions—into the NMT decoding process.
However, external information is rarely perfect. A dictionary might provide a translation that does not fit the current context, or an SMT system might suggest a word that belongs elsewhere in the sentence. Existing "Reading-Fusion" models often lack the judgment to ignore such "noise." If the external information is too noisy, it can actually degrade translation quality compared to a baseline model with no external help.
Methodology
The proposed Reading-Fusion framework with Noise Discrimination categorizes noise into two types:
- Global Noise: Words in the external set that do not belong in the target sentence at all (e.g., an incorrect dictionary sense).
- Local Noise: Words that are relevant to the sentence but irrelevant at the current time step (e.g., a word suggested too early or too late in the sequence).
Key Components
- Global Word Discriminator: Before decoding, this module assigns a "relevance score" to every external word based on the source sentence context.
- Local Word Discriminator: During decoding, this acts as an intelligent gate, deciding whether to use the NMT model's internal prediction or to "copy" from the external word set.
- The
<null>Token: A special "empty" token is added to the external set, providing the attention mechanism a place to "look" when no external suggestions are useful. - Synthetic Training: The authors use a self-supervised approach, creating synthetic datasets by sampling words from reference translations (positive examples) and random vocabulary words (negative examples) to train the discriminators without manual labeling.
Experimental Results
The model was evaluated on Chinese-to-English (Zh-En) using LDC corpora and English-to-German (En-De) using the WMT17 corpus.
Performance Gains
On the Zh-En task, the final model achieved a +4.03 BLEU improvement over the baseline RNNSEARCH model. As shown in the ablation study, each component—global discrimination, local discrimination, and the <null> token—contributes significantly to the overall performance.
Robustness and Versatility
A key insight is the model's robustness; even when external information was 90% noise, the proposed framework remained stable by successfully ignoring irrelevant inputs. Furthermore, the method proved architecture-agnostic, providing a +2.81 BLEU boost when applied to the Transformer architecture. The model also demonstrated versatility across various external information types, including SMT outputs, lexical tables, and bilingual dictionaries.
Conclusion
By distinguishing between global and local noise, this framework allows NMT models to safely leverage imperfect external information. While the current approach treats external data as a "bag of words," future work may adapt these noise discrimination techniques for sequentially encoded information to better utilize full sentences from translation memories or other MT systems.
Summary updated