A topic based document relevance ranking model

Yang Gao, Yue Xu, Yuefeng LiWWW 2014 · DOI

Abstract

Information Filtering (IF) systems often struggle to balance broad thematic understanding with the granular precision required to identify truly relevant documents. This paper introduces the Pattern Enhanced Topic Model (PETM), a novel framework that bridges the gap between Latent Dirichlet Allocation (LDA) and frequent pattern mining. By extracting "Maximum Matched Patterns" from topical document transactions, PETM captures discriminative phrases that signal high relevance. Experimental evaluations on the RCV1 benchmark demonstrate that PETM significantly outperforms traditional term-based and pattern-based models, achieving substantial improvements in precision and F1 scores.

Introduction

Information Filtering (IF) aims to surface relevant documents from massive data streams based on specific user interests. Historically, this field has been dominated by two approaches: term-based models (e.g., BM25) and pattern-based models. Term-based models often suffer from "semantic confusion" due to polysemy and synonymy, while traditional pattern-based models typically assume a single-topic interest, failing to capture the multi-faceted nature of real-world information needs.

While Topic Modeling (specifically LDA) addressed the multi-topic problem, it introduced "blurriness" by representing topics as simple distributions of individual words. The authors propose PETM to combine the multi-topic breadth of LDA with the semantic precision of pattern mining, creating a more robust relevance ranking system.

Methodology

The Pattern Enhanced Topic Model (PETM) operates through a multi-stage pipeline designed to refine thematic signals into actionable ranking features:

  1. Topical Transaction Construction: Standard LDA is applied to a document collection. Words assigned to the same topic within a document are grouped into "topical document transactions."
  2. Pattern Mining & Equivalence Classes: Frequent pattern mining is applied to these transactions to identify "closed patterns." These are grouped into Equivalence Classes—sets of patterns that appear with identical frequency but vary in length.
  3. Maximum Matched Patterns (MMP): During ranking, the model identifies the longest, most specific pattern in an equivalence class present in the document. This "Maximum Matched Pattern" serves as a stronger signal of relevance than generic, shorter word sequences.
  4. Relevance Ranking: The final relevance score is a weighted calculation integrating the document's topic distribution, the statistical significance of the patterns, and a specificity factor derived from pattern length.

Experimental Evaluation

The model was evaluated using the Reuters Corpus Volume 1 (RCV1), a standard benchmark in information retrieval. The researchers utilized the first 50 collections within RCV1, splitting each into a training set to model user interests and a testing set to evaluate ranking performance.

Results

PETM was compared against three baselines: BM25 (term-based), FCP (frequent closed pattern), and PTM (a previous pattern-based topic model). As shown in the performance table below, PETM consistently outperformed all baselines across every major metric.

Performance comparison of PETM against baseline models on the RCV1 dataset, showing significant improvements across all metrics.

Key performance gains included:

  • Top20 Precision: 21.9% improvement over the next best model.
  • Break-even Point (b/p): 28.9% improvement.
  • Mean Average Precision (MAP): 15.5% improvement.
  • F1 Measure: 9.8% improvement.

Discussion and Conclusion

The success of PETM highlights the importance of specificity in relevance ranking. By utilizing Maximum Matched Patterns, the model effectively "de-noises" LDA outputs, moving away from ambiguous "bags of words" toward structured, meaningful phrases. This hybrid granularity allows the model to understand both the general thematic context and the detailed discriminative patterns within those themes.

Future work may address the computational intensity of pattern mining on large-scale streams and explore dynamic tuning of the specificity parameters to adapt to evolving user interests over time.

Summary updated

BibTeX
@inproceedings{gao2014topic,
  author = {Yang Gao and Yue Xu and Yuefeng Li},
  title = {{A topic based document relevance ranking model}},
  booktitle = {The Web Conference},
  year = {2014},
  doi = {10.1145/2567948.2577289},
}
Figure