Investigating the relationship between language model perplexity and IR precision-recall measures

Leif Azzopardi, Mark A. Girolami, Keith van RijsbergenSIGIR 2003 · DOI

Abstract

This paper investigates whether perplexity, the standard metric for evaluating language models, is a reliable predictor of success in Information Retrieval (IR). By testing Probabilistic Latent Semantic Indexing (PLSI) across several datasets, the study finds that while a relationship exists, lower perplexity does not always guarantee better search results. The findings suggest that the optimal model for linguistic prediction may differ from the optimal model for search relevance.

Introduction

In the early 2000s, Language Modeling (LM) became a dominant paradigm for Information Retrieval. In Natural Language Processing, the "gold standard" for a model is perplexity: a measure of how well a model predicts a sample of text. A lower perplexity means the model is less "surprised" by the data.

However, IR researchers primarily focus on Precision and Recall (P-R)—how accurately a system finds relevant documents for a user's query. This research addresses a critical gap: models that achieve a better fit to the data (lower perplexity) do not always yield better search results. The study asks: if massive computational effort is spent to lower a model's perplexity, will the search engine actually improve?

Methodology

The researchers employed Probabilistic Latent Semantic Indexing (PLSI), a latent variable model that represents documents as mixtures of "topics." To test the relationship between perplexity and IR performance, they:

  1. Varied Model Complexity: Adjusted the number of latent variables ($k$) from 2 to 128.
  2. Measured Perplexity: Held out 10% of each document collection as an "out-of-sample" set to calculate how well the model predicted unseen data.
  3. Measured IR Performance: Used the models to index the remaining 90% and calculated the Average Precision (AP) across standard query sets.
  4. Statistical Validation: Used ANOVA and Least Significant Difference (LSD) tests to determine if performance changes were statistically significant.

Datasets

The study used four classic IR benchmark collections:

  • Medline: 1,033 medical abstracts.
  • Cranfield: 1,499 aeronautics documents.
  • CACM: 3,204 computer science documents.
  • CISI: 1,460 library science documents.

Key Results and Analysis

The authors confirmed that PLSI-based models generally outperformed traditional baselines. For example, on the Medline collection, the PLSI-Q model achieved 55.7% Average Precision compared to 51.2% for the Cosine TF-IDF baseline.

However, the visual data reveals a divergence between the two metrics. While the perplexity curves (represented by square symbols) often trend downward as $k$ increases, the Average Precision curves (represented by diamond symbols) typically peak early and then flatten out or decline.

Perplexity and Average Precision for MedLINE and CACM collections.
Perplexity and Average Precision for CISI and Cranfield collections.

Key Insights

  • The "Sweet Spot" for Complexity: The best IR performance was typically achieved when the number of latent variables ($k$) was relatively low—between 8 and 64. Increasing $k$ further continued to lower perplexity but caused IR performance to plateau.
  • Diminishing Returns: The effort required to achieve lower perplexity scores often resulted in negligible gains in search relevance. In many cases, the statistical difference in Average Precision between a "good" model and a "great" model (in terms of perplexity) was not significant.
  • Metric Misalignment: A model that is a perfect "predictor" of text may overfit the specific vocabulary of a collection in a way that hurts its ability to generalize for search queries.

Conclusion and Future Work

The study concludes that while perplexity is a useful guide, it should not be the only metric used when developing models for retrieval. Superior predictive language models do not necessarily provide superior IR performance.

Limitations include the small, homogeneous nature of the corpora used. Future work should investigate these relationships in larger, more diverse document collections and explore different types of language models, such as Dirichlet-based models, to determine if these findings hold across broader contexts.

Summary updated

BibTeX
@inproceedings{azzopardi2003investigating,
  author = {Leif Azzopardi and Mark A. Girolami and Keith van Rijsbergen},
  title = {{Investigating the relationship between language model perplexity and IR precision-recall measures}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2003},
  doi = {10.1145/860435.860505},
}
Figure