An Empirical Investigation of Discounting in Cross-Domain Language Models

Greg Durrett, Dan KleinACL 2011 · OpenReview

Abstract

We investigate the empirical behavior of n-gram discounts within and across domains. When a language model is trained and evaluated on two corpora from exactly the same domain, discounts are roughly constant, matching the assumptions of modified Kneser-Ney LMs. However, when training and test corpora diverge, the empirical discount grows essentially as a linear function of the n-gram count. We adapt a Kneser-Ney language model to incorporate such growing discounts, resulting in perplexity improvements over modified Kneser-Ney and Jelinek-Mercer baselines.

Introduction

In N-gram language modeling, "discounting" is a fundamental technique used to handle the sparsity of training data. By subtracting a small amount from observed counts, models can reserve probability mass for unseen sequences. For decades, Modified Kneser-Ney (MKN) smoothing has been the standard, operating on the assumption that the discount should be a constant value regardless of the N-gram's frequency in the training set.

This assumption holds well when training and test data are drawn from the same distribution. However, modern NLP frequently involves cross-domain scenarios—for instance, applying a model trained on 1990s news to 2000s data. This research investigates whether the "constant discount" rule remains valid as domains diverge.

Empirical Observations of Growing Discounts

The authors calculated the average empirical discount by comparing N-gram counts in training sets to their frequencies in test sets of equal size. If an N-gram appears $k$ times in training and $j$ times in test, the empirical discount is $k - j$.

Figure 1: Average empirical discount as a function of training count. In-domain data (NYT95') shows a flat trend, while cross-domain pairs exhibit linear growth.

As shown in Figure 1, the constant discount assumption only holds for perfectly matched domains (e.g., NYT95'). As soon as the corpora diverge—whether by time or source—the discount begins to grow linearly with the N-gram count. This "regression to the mean" occurs because high-frequency items in one domain (such as specific trending topics or names) are unlikely to maintain that same high frequency in a different context.

The Growing Discount Language Model (GDLM)

To address this, the researchers introduced the Growing Discount Language Model (GDLM). Instead of fixed parameters, they fit a parametric function to the discount: $$f(x) = a + bx^c$$ where $x$ is the N-gram count. A simplified version, GDLM-LIN (where $c=1$), was also evaluated. These models allow the discount to scale with frequency, better capturing the behavior observed in cross-domain data.

Results and Performance

The GDLM was evaluated using the English Gigaword corpus, comparing performance across different years and news agencies (NYT and AFP).

Table 1: Perplexity results comparing GDLM variants against standard baselines (MKN, KN, JM) across different training sets and vocabulary sizes.

As summarized in Table 1, the GDLM and its linear variant consistently outperformed Modified Kneser-Ney (MKNLM) in cross-domain tasks. For example, when training on NYT and testing on AFP, GDLM achieved significant perplexity reductions (lower is better). Crucially, in-domain performance remained stable, as the model naturally reverts to a constant discount when the data overlap is high.

Conclusion

This investigation reveals that the traditional constant-discount assumption is a special case of a more general "growing discount" phenomenon. By allowing discounts to grow linearly with N-gram counts, the GDLM provides a more robust framework for cross-domain language modeling. While neural models have largely replaced N-grams in many applications, the principle that frequency estimates require domain-sensitive adjustment remains a vital insight for modern domain adaptation.

Summary updated

Abstract

We investigate the empirical behavior of n-gram discounts within and across domains. When a language model is trained and evaluated on two corpora from exactly the same domain, discounts are roughly constant, matching the assumptions of modified Kneser-Ney LMs. However, when training and test corpora diverge, the empirical discount grows essentially as a linear function of the n-gram count. We adapt a Kneser-Ney language model to incorporate such growing discounts, resulting in perplexity improvements over modified Kneser-Ney and Jelinek-Mercer baselines.
BibTeX
@inproceedings{durrett2011empirical,
  author = {Greg Durrett and Dan Klein},
  title = {{An Empirical Investigation of Discounting in Cross-Domain Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2011},
  url = {https://openreview.net/forum?id=r1bdsnlOWB},
}

Loading related content...

Figure