A Neural Probabilistic Language Model

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian JanvinJMLR 2003 · OpenReview

Abstract

This foundational 2003 paper introduces the concept of using neural networks to learn distributed representations of words alongside a statistical language model. By mapping discrete words to continuous, real-valued vectors, the model overcomes the "curse of dimensionality" that plagues traditional n-gram models, allowing it to generalize to unseen word sequences based on semantic and syntactic similarity.

Introduction

Traditional statistical language models, such as n-grams, predict the next word in a sequence based on the exact sequence of the previous $n-1$ words. This approach suffers heavily from the curse of dimensionality. Because natural language has a massive vocabulary, the number of possible word combinations grows exponentially as the context window increases.

If a specific sequence of words was never seen in the training data, standard models struggle to assign it a meaningful probability. For example, if the model sees "The cat is walking in the bedroom" during training, it learns that exact sequence. However, it cannot transfer that knowledge to a semantically identical sentence like "A dog was running in a room" because the discrete word tokens are completely different.

Semantic similarity examples illustrating the generalization challenge.

The authors sought a way to share probability mass among "similar" words to vastly improve generalization, leveraging the idea that a small change in the feature space should induce a small change in the predicted probability.

Methodology

The authors propose a feed-forward neural network that learns a joint probability function of word sequences. The architecture is split into two main components that are trained simultaneously:

  1. Distributed Word Features: Every word in the vocabulary $V$ is mapped to a continuous, real-valued feature vector of dimension $m$ (represented by a shared matrix $C$).
  2. Probability Function: A neural network takes the concatenated feature vectors of the previous $n-1$ words, passes them through a hidden layer (with a hyperbolic tangent activation), and outputs a probability distribution for the next word using a softmax layer.
Architecture of the neural probabilistic language model.

Crucially, the word feature vectors and the network weights are trained jointly via stochastic gradient ascent to maximize the penalized log-likelihood of the training data. Because computing the softmax output over tens of thousands of vocabulary words is computationally intensive, the authors implemented a parameter-parallel processing strategy across a cluster of CPUs.

Experimental Setup

The model was evaluated on two text corpora:

  • Brown Corpus: A stream of ~1.18 million words. The vocabulary was capped at 16,383 words.
  • Associated Press (AP) News (1995-1996): A larger dataset containing ~14 million words for training. The vocabulary was capped at 17,964 words.

Key Results and Insights

The neural network significantly outperformed traditional n-gram models in terms of test perplexity:

  • Brown Corpus: The neural network achieved a perplexity of 252, outperforming the best smoothed n-gram model (312) by about 24%.
  • AP News Corpus: The model achieved a perplexity of 109, beating the best n-gram (Modified Kneser-Ney, 117) by about 8%.

Key insights from the study include:

  • Generalization through Geometry: By representing words as continuous vectors, a sentence learned about a "cat" automatically increases the probability of a similar sentence about a "dog" if their vectors are close.
  • Linear Scaling with Context: In traditional n-gram models, parameters scale exponentially with the context window size ($V^n$). In this neural model, parameters scale linearly with context size $n$, allowing for longer context windows without data sparsity issues.
  • Ensembles: The authors found that averaging the predictions of the neural model with an interpolated trigram model consistently yielded the best overall performance.

Conclusion and Future Work

While the model provides superior generalization, it is computationally expensive to train, primarily due to the softmax normalization over the entire vocabulary. The authors suggested several future directions, including speeding up the output layer using tree-structured representations (hierarchical softmax), using recurrent neural networks (RNNs) to capture longer-term context, and assigning multiple vectors to polysemous words to capture different word senses. This work established the foundation for modern word embeddings and deep learning in natural language processing.

Summary updated

BibTeX
@article{bengio2003neural,
  author = {Yoshua Bengio and Réjean Ducharme and Pascal Vincent and Christian Janvin},
  title = {{A Neural Probabilistic Language Model}},
  journal = {J. Mach. Learn. Res.},
  year = {2003},
  url = {https://openreview.net/forum?id=HAMEUQyUHg5},
}
Figure