Automatic Construction of Evaluation Sets and Evaluation of Document Similarity Models in Large Scholarly Retrieval Systems

Kriste Krstovski, David A. Smith, Michael J. KurtzScholarly Big Data @ AAAI 2016

TL;DR

This paper introduces a novel, efficient method for automatically evaluating document similarity models in large scholarly retrieval systems. By leveraging user download logs, they create "pseudo-relevant" document pairs, bypassing the costly human annotation process typically required for traditional evaluation. Their proposed "random histogram slope analysis" (rHSA) method shows a strong correlation with established metrics like Mean Average Precision (MAP), offering a practical solution for researchers and developers.

What Problem Does This Paper Address?

Evaluating document similarity models, crucial for tasks like recommendation and exploratory data analysis in scholarly retrieval systems, traditionally relies on human-annotated "test collections." These collections consist of query documents and sets of human-judged relevant (similar) documents. Creating these sets is a labor-intensive and expensive process, especially for large-scale systems that cover broad scientific disciplines. This bottleneck makes it difficult for developers to quickly assess and compare different similarity models, particularly when starting a new project or dealing with a new document collection. The paper aims to bridge this gap by proposing an automated method for generating evaluation sets and a corresponding evaluation metric.

Methods

The core of the paper's contribution lies in two main areas:

  • Automatic Evaluation Set Construction: Instead of human annotators, the authors utilize user download logs from scholarly IR systems. They hypothesize that documents downloaded consecutively within a specific time window (e.g., 10 seconds to 1 hour) are likely to be "pseudo-relevant" to each other. This is analogous to how clickstream data is used in web search, where a user clicking a link is assumed to be more relevant than a random click. They contrast these "consecutively downloaded document pairs" with "randomly generated document pairs" to form their evaluation sets.
  • Random Histogram Slope Analysis (rHSA): This is the novel evaluation metric proposed. It works by:
  • Histogram Generation: Computing histograms of similarity scores for both the consecutively downloaded (pseudo-relevant) pairs and the randomly generated (dissimilar) pairs, using a chosen document similarity model.
  • Log Division: Dividing the two histograms in log space. This operation smooths out differences in the total number of pairs and focuses on the distribution of similarity values.
  • Slope Calculation: Computing the slope of the resulting log division curve. The authors hypothesize that a steeper slope indicates a better-performing model, as it suggests that similar documents are more tightly clustered in the model's representation space. This slope is then used as the evaluation score.

Datasets

The authors utilized download logs from the SAO/NASA Astrophysics Data System (ADS) for the past three years, focusing on articles from the Astrophysical Journal (ApJ). They extracted approximately 23 million downloads, resulting in 1.7 million unique downloads. From these logs, they created two primary test sets:

  • 1M Set: Consisting of one million consecutively downloaded document pairs and one million randomly generated document pairs.
  • 100q Set: A smaller set derived from documents with at least 100 succeeding downloads, used for traditional IR evaluation (MAP) to compare against rHSA.

Key Insights and Ideas

  • User Behavior as a Proxy for Similarity: The fundamental insight is that user download patterns, specifically consecutive downloads, can serve as a reliable proxy for document similarity, effectively creating "pseudo-relevant" sets without human intervention.
  • Histogram Shape as a Performance Indicator: The distribution of similarity scores between similar and dissimilar document pairs, when visualized as histograms, reveals distinct shapes. A good similarity model should push similar documents closer together (higher similarity scores) and dissimilar documents further apart (lower similarity scores). rHSA leverages this difference.
  • Efficiency of rHSA: rHSA is significantly more efficient than traditional MAP evaluation because it doesn't require traversing ranked lists for each query. This makes it ideal for large-scale collections.
  • Predictive Power of rHSA: The paper demonstrates that rHSA scores have a very high linear and rank correlation with MAP scores, suggesting that rHSA can effectively predict the performance of a document similarity model.

Key Results

  • High Correlation with MAP: rHSA achieved very high linear (R) and rank (ρ) correlation coefficients with MAP across various tf-idf and LDA model configurations and across both the 1M and 100q evaluation sets. For instance, correlations for the 1M set were often above 0.97.
  • Model Ranking Consistency: rHSA consistently ranked models in a similar order to MAP. For LDA models, rHSA identified T=5k as the best configuration, aligning with MAP results. For tf-idf models, rHSA favored the configuration using all tokens.
  • Efficiency: The paper highlights that rHSA is more efficient to compute than traditional IR metrics like MAP.

Limitations and Future Work

  • Heuristics for Pseudo-Relevance: The paper uses heuristics (time windows of 10 seconds to 1 hour) to define consecutively downloaded pairs. The authors acknowledge that these are heuristics and might not perfectly capture all instances of true similarity or exclude all irrelevant pairs.
  • Domain Specificity: The study focuses on scholarly articles from the Astrophysical Journal. While the approach is generalizable, its performance might vary across different domains or types of scholarly literature.
  • Future Work: The authors suggest further exploration of different similarity metrics and document representations, as well as investigating the impact of various log mining techniques on the quality of pseudo-relevant sets. They also implicitly suggest that rHSA could be integrated into active learning loops for model refinement.

Summary updated

BibTeX
@inproceedings{krstovski2016automatic,
  author = {Kriste Krstovski and David A. Smith and Michael J. Kurtz},
  title = {{Automatic Construction of Evaluation Sets and Evaluation of Document Similarity Models in Large Scholarly Retrieval Systems}},
  booktitle = {Scholarly Big Data @ AAAI},
  year = {2016},
}
Figure