Learning to Discriminate Noises for Incorporating External Information in Neural Machine Translation

Zaixiang Zheng, Shujian Huang, Zewei Sun, Rongxiang Weng, Xin-Yu Dai, Jiajun ChenarXiv 2018 · OpenReview

Abstract

While external information like dictionaries or suggestions from other translation systems can boost Neural Machine Translation (NMT) performance, this data is often riddled with "noise" that can mislead the model. This paper introduces a dual-discriminator framework that identifies and filters out irrelevant external words at both the sentence level (global noise) and the specific word-generation step (local noise), making NMT significantly more robust to low-quality external inputs.

Introduction

Neural Machine Translation has become the industry standard, but it often struggles with rare words or domain-specific terminology. To address this, researchers frequently "inject" external information—such as bilingual dictionaries, outputs from older Statistical Machine Translation (SMT) systems, or human suggestions—into the NMT decoding process.

However, external information is rarely perfect. A dictionary might provide a translation that does not fit the current context, or an SMT system might suggest a word that belongs elsewhere in the sentence. Existing "Reading-Fusion" models often lack the judgment to ignore such "noise." If the external information is too noisy, it can actually degrade translation quality compared to a baseline model with no external help.

Methodology

The proposed Reading-Fusion framework with Noise Discrimination categorizes noise into two types:

  1. Global Noise: Words in the external set that do not belong in the target sentence at all (e.g., an incorrect dictionary sense).
  2. Local Noise: Words that are relevant to the sentence but irrelevant at the current time step (e.g., a word suggested too early or too late in the sequence).
Architecture of the Reading-Fusion framework with Noise Discrimination, illustrating the global and local discriminators.

Key Components

  • Global Word Discriminator: Before decoding, this module assigns a "relevance score" to every external word based on the source sentence context.
  • Local Word Discriminator: During decoding, this acts as an intelligent gate, deciding whether to use the NMT model's internal prediction or to "copy" from the external word set.
  • The <null> Token: A special "empty" token is added to the external set, providing the attention mechanism a place to "look" when no external suggestions are useful.
  • Synthetic Training: The authors use a self-supervised approach, creating synthetic datasets by sampling words from reference translations (positive examples) and random vocabulary words (negative examples) to train the discriminators without manual labeling.

Experimental Results

The model was evaluated on Chinese-to-English (Zh-En) using LDC corpora and English-to-German (En-De) using the WMT17 corpus.

Performance Gains

On the Zh-En task, the final model achieved a +4.03 BLEU improvement over the baseline RNNSEARCH model. As shown in the ablation study, each component—global discrimination, local discrimination, and the <null> token—contributes significantly to the overall performance.

Ablation study on the Zh-En task showing the incremental improvements of each noise discrimination component.

Robustness and Versatility

A key insight is the model's robustness; even when external information was 90% noise, the proposed framework remained stable by successfully ignoring irrelevant inputs. Furthermore, the method proved architecture-agnostic, providing a +2.81 BLEU boost when applied to the Transformer architecture. The model also demonstrated versatility across various external information types, including SMT outputs, lexical tables, and bilingual dictionaries.

Conclusion

By distinguishing between global and local noise, this framework allows NMT models to safely leverage imperfect external information. While the current approach treats external data as a "bag of words," future work may adapt these noise discrimination techniques for sequentially encoded information to better utilize full sentences from translation memories or other MT systems.

Summary updated

Abstract

Previous studies show that incorporating external information could improve the translation quality of Neural Machine Translation (NMT) systems. However, there are inevitably noises in the external information, severely reducing the benefit that the existing methods could receive from the incorporation. To tackle the problem, this study pays special attention to the discrimination of the noises during the incorporation. We argue that there exist two kinds of noise in this external information, i.e. global noise and local noise, which affect the translations for the whole sentence and for some specific words, respectively. Accordingly, we propose a general framework that learns to jointly discriminate both the global and local noises, so that the external information could be better leveraged. Our model is trained on the dataset derived from the original parallel corpus without any external labeled data or annotation. Experimental results in various real-world scenarios, language pairs, and neural architectures indicate that discriminating noises contributes to significant improvements in translation quality by being able to better incorporate the external information, even in very noisy conditions.
BibTeX
@misc{zheng2018learning,
  author = {Zaixiang Zheng and Shujian Huang and Zewei Sun and Rongxiang Weng and Xin-Yu Dai and Jiajun Chen},
  title = {{Learning to Discriminate Noises for Incorporating External Information in Neural Machine Translation}},
  year = {2018},
  eprint = {1810.10317},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.10317},
}
Figure