Investigating the relationship between language model perplexity and IR precision-recall measures

Leif Azzopardi, Mark Girolami, Keith van RisjbergenSIGIR 2003 · DOI

TL;DR

This paper investigates whether a language model's perplexity (a measure of its predictive uncertainty) directly correlates with its effectiveness in information retrieval (IR). The authors found a systematic relationship, but it wasn't statistically significant, suggesting that simply minimizing perplexity might not be the sole path to superior IR performance.

What Problem Does This Paper Address?

In Information Retrieval (IR), language models (LMs) are a cornerstone for understanding document relevance. A key metric for evaluating LMs is "perplexity," which quantifies how well a model predicts a sequence of words. Intuitively, a lower perplexity suggests a better, more confident model. However, the direct impact of this improved perplexity on the actual performance of IR systems, measured by precision and recall, remained an open question. This paper aims to bridge this gap by empirically examining if LMs with lower perplexity consistently lead to better precision-recall scores in IR tasks.

Methods

The authors employed a probabilistic latent semantic indexing (PLSI) model, a type of aspect-based language model known for its strong performance in both perplexity and IR. PLSI has previously shown reduced perplexity compared to smoothed unigram models and superior precision-recall figures. The experiments involved using four standard IR collections (Medline, Cranfield, CACM, CISI). For each collection, they created out-of-sample sets for evaluation and used the remainder for model training. The PLSI model was estimated with varying numbers of latent variables (ranging from 2 to 128) to explore how this parameter affects both perplexity and average precision. They used a standard approach for model-based indexing (PLSI-U and PLSI-Q) and computed perplexity as detailed in prior work.

Datasets

The study utilized four publicly available text collections commonly used in IR research:

  • Medline: 1033 documents, 5134 terms.
  • Cranfield: 1499 documents, 3262 terms.
  • CACM: 3204 documents, 4326 terms.
  • CISI: 1460 documents, 3918 terms.

These collections were preprocessed by removing standard stop words, applying Porter Stemming, and then removing infrequent terms (appearing two times or less).

Key Insights and Ideas

  • Perplexity isn't everything for IR: While a systematic relationship between lower perplexity and higher average precision was observed, it wasn't statistically significant. This implies that optimizing solely for perplexity might not guarantee the best IR results.
  • The "sweet spot" for latent variables: For the latent variable approach used (PLSI), best performance in terms of average precision was achieved with a relatively low number of latent variables, typically between 8 and 64. This suggests that overly complex models (with too many latent variables) might not necessarily translate to better retrieval.
  • Collection dependency: The relationship between perplexity and precision-recall wasn't uniform across all datasets. The CACM collection, in particular, seemed "resistant to reductions in perplexity," indicating that the nature of the corpus can influence how well LM metrics predict IR performance.

Key Results

The paper presents results in Table 1, showing the best performance achieved by different models across the four collections. The PLSI models (PLSI-U and PLSI-Q) generally outperformed Cosine TF-IDF. Figures 1 and 2 visually depict the relationship between average precision and perplexity as the number of latent variables increases. For instance, in the Medline collection, as the number of latent variables increases, perplexity generally decreases, and average precision initially increases before plateauing or slightly decreasing. While ANOVA tests showed significant differences in means for both perplexity and AP across collections, follow-up LSD tests revealed that only a small subset of models showed statistically significant differences in mean AP.

Limitations and Future Work

The authors acknowledge that their study was conducted on relatively small, homogeneous corpora. They suggest that further investigation on larger and more heterogeneous document collections is warranted. Additionally, they propose exploring other types of language models beyond the aspect-based PLSI model to understand the broader relationship between LM perplexity and IR performance. Future work will also investigate matching functions that explicitly define query-document probabilities based on specific LMs.

Summary updated

Abstract

An empirical study has been conducted investigating the relationship between the performance of an aspect based language model in terms of perplexity and the corresponding information retrieval performance obtained. It is observed, on the corpora considered, that the perplexity of the language model has a systematic relationship with the achievable precision recall performance though it is not statistically significant.
BibTeX
@inproceedings{azzopardi2003investigating,
  author = {Leif Azzopardi and Mark Girolami and Keith van Risjbergen},
  title = {{Investigating the relationship between language model perplexity and IR precision-recall measures}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2003},
  doi = {10.1145/860500.860505},
}
Figure