A Dynamic Language Model Based on Individual Word Domains

E.i. Sicilia-Garcia, Ji Ming, F. J. SmithCOLING 2000

TL;DR

This paper introduces a novel dynamic language model that improves performance by creating individual language models for "significant" words and combining them with a global model. This approach, particularly using a "probabilistic-union" combination method, significantly reduces perplexity, a key metric for language model quality, by better capturing word context.

What Problem Does This Paper Address?

Statistical language models are crucial for understanding and generating human language. They work by estimating the probability of a word given its preceding context. Traditional models, like n-grams, struggle with capturing the nuanced meaning and context of individual words, especially when dealing with rare word sequences or when a word's significance is tied to specific domains. Dynamic language models attempt to address this by adapting probabilities based on recent word history, but this paper aims to go further by creating specialized models for words that carry significant meaning within a text. The challenge is how to effectively combine these specialized models with a general language model to achieve better overall performance.

Methods

The core innovation is the creation of individual word language models. For each "significant" word (defined as non-stop words that contribute to the text's content), a separate language model is trained using a corpus specifically extracted from the global training data where that significant word appears. This creates smaller, more focused corpora for these individual models.

The paper then proposes a novel method for combining these individual models with a global n-gram model: the probabilistic-union model. This method is inspired by techniques used in noisy speech recognition to mitigate the impact of zero or near-zero probabilities that can arise from sparse data in individual models. Instead of simple averaging or weighted sums, the probabilistic-union model uses a logical union of conjunctions, essentially a sum of products, to combine probabilities. This approach is designed to reduce the negative impact of zero probabilities from individual word models.

Datasets

The paper uses the Wall Street Journal (WSJ) corpus for training and evaluation. Specifically, they use a portion of the WSJ03 corpus for training and a subset of the WSJ0 test file for evaluation. The training corpus contains approximately 9.5 million tokens.

Key Insights and Ideas

  • Individual word domains are powerful: Creating specialized language models for significant words, trained on context-specific data, leads to more precise probability estimates than a single global model.
  • The "zeroing effect" is a problem: When combining models, especially with sparse data, zero probabilities from individual models can disproportionately reduce the overall probability.
  • Probabilistic-union offers a solution: This novel combination method effectively mitigates the "zeroing effect" by using a probabilistic sum (⊕) that is less sensitive to individual zero probabilities.
  • Context matters: The paper explores different ways to define the context for individual word models, including paragraph context, and also investigates exponential decay to give more weight to recent word occurrences.

Key Results

The paper reports significant improvements in perplexity, a measure where lower is better.

  • A simple weighted combination of models achieved up to a 10% improvement.
  • An exponential decay model (Eq. 9) showed improvements of over 17%.
  • The probabilistic-union model achieved the best results, with an improvement of 20% when using a maximum of 6 word models and an order of 5 (sum of products of pairs).

Limitations and Future Work

  • Exploration of Probabilistic-Union is ongoing: The authors acknowledge that the probabilistic-union model has "yet to be fully explored" and that the initial results, while promising, are preliminary.
  • Computational cost: The process of normalisation for combining models can be slow, leading them to reduce n-gram sizes for experiments.
  • Further experiments planned: The authors intend to conduct further experiments with larger training corpora and test files to validate their findings.
  • Sophistication of combination: They suggest that more sophisticated combination models are likely to offer further performance gains beyond the methods explored.

Summary updated

BibTeX
@inproceedings{siciliagarcia2000dynamic,
  author = {E.I. Sicilia-Garcia and Ji Ming and F. J. Smith},
  title = {{A Dynamic Language Model Based on Individual Word Domains}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2000},
}
Figure