On Using Very Large Target Vocabulary for Neural Machine Translation

Sébastien Jean, Kyunghyun Cho, Roland Memisevic, Yoshua Bengio

This seminal 2015 paper from the Montreal Institute for Learning Algorithms (MILA) addresses a fundamental bottleneck in early Neural Machine Translation (NMT): the "Large Vocabulary Problem." While traditional NMT models were constrained by computational limits to small vocabularies, this work introduced techniques to scale to over 500,000 words, significantly improving translation quality.

The Challenge: The Large Vocabulary Problem

In early NMT architectures, the final layer typically uses a softmax function to predict the next word in a sequence. Computing this probability requires a normalization constant calculated across the entire target vocabulary. Because this calculation scales linearly with vocabulary size, it becomes computationally prohibitive beyond 30,000 to 50,000 words.

To manage this, researchers historically used a "shortlist" of the most frequent words, mapping all others to a single [UNK] (unknown) token. This approach suffered from two major flaws:

  1. Information Loss: The model could not translate rare words, proper nouns, or technical terminology.
  2. Performance Degradation: Translation quality, measured by BLEU scores, dropped sharply as the number of out-of-vocabulary words in a source sentence increased.

Methodology: Scaling to 500,000 Words

The authors proposed a two-pronged approach to handle very large vocabularies (LV) without a corresponding explosion in computational cost.

1. Training with Importance Sampling

Instead of calculating the full softmax over 500,000 words for every parameter update, the authors employed importance sampling. They partitioned the training data and, for each partition, created a small subset of the vocabulary containing only the words present in that specific batch. By updating weights only for this subset, the training complexity became independent of the total vocabulary size, allowing for efficient GPU optimization through matrix-matrix multiplications.

2. Decoding with Candidate Lists

At test time, searching through a 500,000-word output layer for every generated token is inefficient. To solve this, the authors introduced candidate lists. For a given source sentence, the model selects a small set of likely target words, including the top-$K$ most frequent words and potential translations derived from a pre-computed alignment dictionary.

Figure 1: BLEU score performance as a function of the candidate list size (K'). The plot demonstrates that performance stabilizes quickly and is further enhanced by UNK replacement techniques.

Experimental Results

The models were evaluated on the WMT’14 benchmarks for English→French and English→German. The results demonstrated that the Large Vocabulary (LV) approach consistently outperformed standard NMT baselines and rivaled established phrase-based statistical machine translation (SMT) systems.

Key Performance Metrics

On the English→French task, the proposed "RNNsearch-LV" model achieved significant gains over the baseline. As shown in the results table below, the addition of the candidate list and UNK replacement strategies allowed the model to surpass previous state-of-the-art results from Google.

Figure 4: BLEU scores on the WMT'14 English-to-French task. RNNsearch-LV consistently outperforms the basic NMT model and the Google baseline, especially when combined with UNK replacement and ensembling.

Efficiency and Speed

Despite the massive increase in vocabulary size, the candidate list approach kept decoding speeds practical. On a GPU, the RNNsearch-LV model with a candidate list required only 0.05s per word, compared to 0.02s for the small-vocabulary baseline. Without the candidate list, decoding with the full 500k vocabulary was five times slower.

Conclusion and Impact

This research proved that NMT could scale to handle the full diversity of human language by borrowing and adapting concepts from importance sampling and traditional SMT alignment. While the field eventually shifted toward subword units (like Byte Pair Encoding) to solve the vocabulary problem more elegantly, this paper remains a landmark for its success in breaking the 30,000-word barrier and pushing NMT toward state-of-the-art performance on complex translation tasks.

Summary updated

BibTeX
@inproceedings{jean2015using,
  author = {Sébastien Jean and Kyunghyun Cho and Roland Memisevic and Yoshua Bengio},
  title = {{On Using Very Large Target Vocabulary for Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/p15-1001},
}
Figure