Optimizing two stage bigram language models for IR

Sara Javanmardi, Jianfeng Gao, Kuansan WangWWW 2010 · OpenReview

Abstract

Although higher order language models (LMs) have shown benefit of capturing word dependencies for Information retrieval(IR), the tuning of the increased number of free parameters remains a formidable engineering challenge. Consequently,in many real world retrieval systems, applying higher order LMs is an exception rather than the rule. In this study, we address the parameter tuning problem using a framework based on a linear ranking model in which different component models are incorporated as features. Using unigram and bigram LMs with 2 stage smoothing as examples, we show that our method leads to a bigram LM that outperforms significantly its unigram counterpart and the well-tuned BM25 model.
BibTeX
@inproceedings{javanmardi2010optimizing,
  author = {Sara Javanmardi and Jianfeng Gao and Kuansan Wang},
  title = {{Optimizing two stage bigram language models for IR}},
  booktitle = {The Web Conference},
  year = {2010},
  url = {https://openreview.net/forum?id=S1V7SZ-dWS},
}

Loading related content...

Figure