Improving Language Modelling with Noise Contrastive Estimation

Farhana Ferdousi Liza, Marek GrzesAAAI 2018 · DOI

Abstract

Noise Contrastive Estimation (NCE) is frequently dismissed as a computationally efficient but statistically inferior alternative to the exact softmax for large-scale language modeling. This paper challenges this view, demonstrating that with proper hyperparameter tuning—specifically a "search-then-converge" learning rate schedule and careful weight initialization—NCE can outperform exact softmax models. On the Penn Tree Bank dataset, an NCE-trained LSTM achieved a state-of-the-art test perplexity of 69.995, surpassing previous single-model benchmarks while reducing training time.

Introduction: The Softmax Bottleneck

In neural language modelling, the standard approach is to use the softmax function to produce a probability distribution over a vocabulary. However, softmax requires calculating a partition function—a sum over every word in the vocabulary—which becomes a massive computational bottleneck as vocabulary sizes grow. Noise Contrastive Estimation (NCE) bypasses this by reframing language modelling as a binary classification task: distinguishing real data from random noise. While NCE is known for its speed, it has historically been viewed as statistically weaker than exact softmax. This research argues that NCE's perceived inferiority stems from a lack of rigorous hyperparameter optimization rather than inherent theoretical flaws.

Methodology: The NCE Training Recipe

The authors implemented a standard LSTM recurrent neural network, replacing the traditional softmax output layer with an NCE layer. Their success relied on a specific training recipe:

  1. Search-then-Converge Schedule: They utilized a learning rate schedule that maintains a high learning rate during a "search" phase to explore the loss landscape before decaying it during a "convergence" phase.
  2. Optimizing the Search Duration ($\tau$): A critical hyperparameter is $\tau$, the duration of the search phase. The authors found that the optimal $\tau$ varies with model size, generally falling between one-third and two-thirds of total training epochs.
  3. Weight Initialization: NCE proved highly sensitive to initial weights. Using a smaller range for uniform distribution initialization than standard Glorot/Xavier methods significantly improved stability.
  4. Noise Sampling: Using 600 noise samples per data point from a power-law distribution provided an optimal balance between speed and accuracy.
Validation perplexity across different search phase durations ($\tau$) for Small, Medium, and Large models.

Results: Surpassing the Gold Standard

The primary benchmark was the Penn Tree Bank (PTB) dataset. Despite its relatively small 10k-word vocabulary (where exact softmax is feasible), it served as a rigorous test for NCE's competitive potential.

  • State-of-the-Art Performance: The "Large" NCE model achieved a test perplexity of 69.995, outperforming the previous best single-model LSTM results (~78.4).
  • Efficiency Gains: The NCE Large model was faster to train, requiring roughly 7.5 hours compared to over 9 hours for an equivalent softmax model.
  • Convergence Behavior: As shown in the training curves, NCE models often reach lower final perplexities than their softmax counterparts when the search phase is appropriately timed.
Convergence of NCE and Softmax models over training epochs, highlighting the impact of the search phase duration $\tau$.

Key Insights and Discussion

NCE as a Global Explorer

A surprising finding is that NCE can find better local optima than softmax. The noisier stochastic gradients inherent in NCE's binary classification task, combined with a long search phase, allow the model to escape poor local minima that often trap standard softmax models.

Robustness to Overfitting

While softmax models typically began to overfit after approximately 20 epochs on PTB, NCE models remained stable or continued to improve. This suggests that NCE may have an inherent regularizing effect, making it more robust during extended training.

Conclusion

This study demonstrates that NCE is not merely a "fast approximation" but a powerful objective function that can exceed the performance of exact softmax when tuned correctly. While NCE is more sensitive to initialization and requires careful scheduling of the search phase, its combination of speed and accuracy makes it a compelling choice for large-scale language modelling. Future work will explore these strategies on billion-word corpora and more modern architectures like Transformers.

Summary updated

BibTeX
@inproceedings{liza2018improving,
  author = {Farhana Ferdousi Liza and Marek Grzes},
  title = {{Improving Language Modelling with Noise Contrastive Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11967},
}
Figure