Shrinking Exponential Language Models

Stanley ChenNAACL 2009

TL;DR

This paper introduces two heuristics for "shrinking" exponential language models, leading to improved performance without sacrificing training accuracy. By analyzing the relationship between training and test set cross-entropy, the authors demonstrate how adding redundant features can paradoxically reduce model size and boost performance, a principle they apply to create novel class-based and domain-adapted language models that outperform existing methods.

What Problem Does This Paper Address?

Language models, particularly those used in speech recognition and natural language processing, are often massive. This size poses challenges for deployment and efficiency. The core problem this paper tackles is how to reduce the size of exponential language models while improving their performance, rather than just maintaining it. Traditional approaches to improving language models often involve adding more features, which typically increases model size and complexity. This work questions that assumption by exploring the trade-off between model size and performance, specifically within the framework of exponential language models.

Methods

The paper's central contribution lies in two heuristics derived from an empirical observation about exponential language models:

  • Heuristic 1: Shrinking through Redundancy. The authors observe that adding "redundant" features to an exponential model can actually decrease the magnitude of its parameter values (specifically, the sum of absolute values of parameters, $\sum |\lambda_i|$). This is achieved by adding features that are sums of existing features. The key insight is that if a model can be "shrunk" (i.e., $\sum |\lambda_i|$ reduced) without increasing its training set cross-entropy, its test set cross-entropy should improve. This principle is applied to:
  • Class-Based Language Models: They propose a novel class-based model (Model M) that leverages word classes to create features that effectively "shrink" n-gram features. Instead of directly modeling word sequences, they introduce features that group words into classes, allowing the model to generalize better and reduce its overall parameter footprint.
  • Backoff Features in N-grams: They analyze how backoff features in n-gram models contribute to performance. They argue that backoff features improve performance not by improving training set accuracy, but by shrinking the model size.
  • Heuristic 2: Domain Adaptation via Priors. This heuristic suggests using a "similar" distribution, estimated from an independent training set, as a prior for the target model. This is applied to domain adaptation by building an out-of-domain language model and using it as a prior for an in-domain model. This approach is shown to be equivalent to the Minimum Discrimination Information (MDI) method for domain adaptation, but with the added benefit of regularization.

Datasets

The experiments were conducted on the following datasets:

  • Wall Street Journal (WSJ) data: Used for training and evaluating language models, including both word-based and class-based models. Various subsets of WSJ data were used for training (1k, 10k, 100k, 900k sentences).
  • Broadcast News data: Used as the out-of-domain data for domain adaptation experiments.
  • CSR-III Text corpus: Used for verbalized punctuation data in speech recognition experiments.

Key Insights and Ideas

  • Shrinking is Good: The core insight is that reducing the "size" of an exponential language model (measured by $\sum |\lambda_i|$) can lead to better test set performance, even if it means adding more features. This is counter-intuitive to the typical goal of minimizing feature count.
  • Backoff Features as Shrinkers: Backoff features in n-gram models are not just about handling unseen n-grams; they actively contribute to model shrinkage, which in turn improves performance.
  • Class-Based Models as Shrinkers: By grouping words into classes, class-based models can create features that effectively shrink the parameter space compared to a purely word-based model, leading to improved performance.
  • Priors for Domain Adaptation: Using an out-of-domain model as a prior for an in-domain model is a powerful technique for domain adaptation, effectively transferring knowledge and shrinking the adapted model.
  • The Power of Regularization: The paper emphasizes the role of regularization (specifically $l_1+l_3$) in achieving these shrinkage effects and improving performance.

Key Results

  • Novel Class-Based Model (Model M): Outperformed baseline word trigram models by 28% in perplexity and 1.9% absolute in word-error rate on WSJ data. It also achieved significant gains over existing class-based models.
  • Domain Adaptation (MDI with Regularization): The proposed MDI approach with regularization outperformed linear interpolation and count merging by significant margins in both perplexity (up to 10% absolute on the smallest data set) and word-error rate (up to 0.7% absolute).
  • Model Size Reduction: In experiments, adding backoff features or class information led to a decrease in the sum of absolute parameter values ($\sum |\lambda_i|$), consistent with the paper's heuristics.

Limitations and Future Work

  • Vagueness of Heuristics: The authors acknowledge that Heuristics 1 and 2 are "quite vague" and that determining when they will be effective remains an open question. Future work could focus on developing more precise methods for identifying suitable features or priors for shrinkage.
  • Feature Selection for MDI: For MDI models, selecting the appropriate feature set is crucial, especially with limited adaptation data. The paper suggests using unigram features intuitively, but more systematic approaches could be explored.
  • Exploring Class Definitions: The paper mentions that choosing word classes to optimize model shrinkage is a potential avenue for future research, but it was not pursued in this work.

Summary updated

BibTeX
@inproceedings{chen2009shrinking,
  author = {Stanley Chen},
  title = {{Shrinking Exponential Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}

Loading related content...

Figure