An Empirical Evaluation of Noise Contrastive Estimation for the Neural Network Joint Model of Translation

Colin CherryNAACL 2016 · OpenReview

Abstract

The neural network joint model of translation or NNJM (Devlin et al., 2014) combines source and target context to produce a powerful translation feature. However, its softmax layer necessitates a sum over the entire output vocabulary, which results in very slow maximum likelihood (MLE) training. This has led some groups to train using Noise Contrastive Estimation (NCE), which side-steps this sum. We carry out the first direct comparison of MLE and NCE training objectives for the NNJM, showing that NCE is significantly outperformed by MLE on large-scale ArabicEnglish and Chinese-English translation tasks. We also show that this drop can be avoided by using a recently proposed translation noise distribution.
BibTeX
@inproceedings{cherry2016empirical,
  author = {Colin Cherry},
  title = {{An Empirical Evaluation of Noise Contrastive Estimation for the Neural Network Joint Model of Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=S1NcZmb_-B},
}

Loading related content...

Figure