Improving a statistical language model by modulating the effects of context words.

Zhang Yuecheng, Andriy Mnih, Geoffrey E. HintonESANN 2008 · OpenReview

Abstract

This paper explores an architectural enhancement to neural language models by introducing a gating mechanism that dynamically weights the importance of context words. By modulating how much each context word contributes to the prediction of the next word based on the surrounding text, the proposed model achieves significant improvements in perplexity over traditional n-gram models and existing neural probabilistic language models.

Introduction

In 2008, the log-bilinear (LBL) model represented a state-of-the-art approach to neural language modeling. The LBL model predicts the next word in a sequence by calculating a linear combination of the feature vectors (embeddings) of the preceding $n-1$ words. However, a fundamental limitation of the standard LBL model is its static treatment of context. It assumes that the influence of a word at a specific relative position is constant, regardless of the actual words present.

In natural language, the relevance of a context word is highly dependent on its surroundings. For instance, a sentence boundary (punctuation) often signals that preceding words are less relevant for predicting the next word. The authors proposed a method to allow the model to "gate" or filter these influences dynamically.

Methods and Architecture

The researchers introduced three primary extensions to the baseline LBL (referred to as Model 0):

  1. Model 1 (The Gating Network): This is the core contribution. A hidden layer processes all context words simultaneously to output a scalar weighting coefficient for each word. This coefficient scales the word's feature vector before it is used for prediction. If the network identifies a word as irrelevant, its influence is effectively suppressed.
  2. Model 2 (Non-linear Addition): Instead of gating, this model uses a hidden layer to compute a general non-linear feature vector that is added to the predicted vector of the target word.
  3. Model 3 (The Hybrid): A combination of both the gating mechanism and the additive non-linear term.
Architecture of the gated log-bilinear model (Model 1), showing how higher-order features modulate context vectors $c_i$ via gating signals $s_i$.

The models were trained using steepest descent to maximize log-likelihood, with word embeddings learned concurrently with the gating parameters.

Experimental Results

The models were evaluated on the APNews (Associated Press News) dataset, utilizing a vocabulary of approximately 18,000 words and a fixed context window of 5 words.

Performance Comparison

The quantitative results demonstrated that the gating mechanism (Model 1) provided the most significant gains. With 500 hidden units, Model 1 achieved a test set perplexity of 104.3, representing a 10.8% reduction over the baseline Model 0. Notably, Model 1 outperformed both the traditional Kneser-Ney 5-gram model and the then-standard Neural Probabilistic Language Model (NPLM) by Bengio et al.

Quantitative results on the APNews dataset comparing various model configurations and baselines.

Key Insights

  • Modulation vs. Prediction: A key finding was that using higher-level features to modulate context words was more effective than using those same features to directly predict the next word. Model 2, despite being more mathematically general, performed worse than the more constrained gating approach of Model 1.
  • Inductive Bias: The success of the gating mechanism suggests that incorporating specific inductive biases—such as forcing the model to reason about the relative importance of inputs—can be more effective than simply increasing raw model capacity.
  • Complexity Trade-offs: Model 3, the hybrid approach, did not yield significant improvements over Model 1 and occasionally performed worse, likely due to increased training difficulty or overfitting.

Conclusion

By introducing a gating mechanism to modulate context word influence, the authors successfully addressed a major rigidity in log-bilinear models. This approach allows the model to implicitly handle structural linguistic elements like punctuation and varying word relevance, setting a new benchmark for neural language modeling performance at the time.

Summary updated

BibTeX
@inproceedings{yuecheng2008improving,
  author = {Zhang Yuecheng and Andriy Mnih and Geoffrey E. Hinton},
  title = {{Improving a statistical language model by modulating the effects of context words.}},
  booktitle = {ESANN},
  year = {2008},
  url = {https://openreview.net/forum?id=sYiZcWnI-IK},
}
Figure