User behavior driven ranking without editorial judgments

Taesup Moon, Georges Dupret, Shihao Ji, Ciya Liao, Zhaohui Zheng

Abstract

We explore the potential of using users click-through logs where no editorial judgment is available to improve the ranking function of a vertical search engine. We base our analysis on the Cumulate Relevance Model, a user behavior model recently proposed as a way to extract relevance signal from click-through logs. We propose a novel way of directly learning the ranking function, effectively by-passing the need to have explicit editorial relevance label for each query-document pair. This approach potentially adjusts more closely the ranking function to a variety of user behaviors both at the individual and at the aggregate levels. We investigate two ways of using behavioral model; First, we consider the parametric approach where we learn the estimates of document relevance and use them as targets for the machine learned ranking schemes. In the second, functional approach, we learn a function that maximizes the behavioral model likelihood, effectively by-passing the need to estimate a substitute for document labels. Experiments using user session data collected from a commercial vertical search engine demonstrate the potential of our approach. While in terms of DCG, the editorial model out-perform the behavioral one, online experiments show that the behavioral model is on par --if not superior-- to the editorial model. To our knowledge, this is the first report in the Literature of a competitive behavioral model in a commercial setting
BibTeX
@inproceedings{moon2010user,
  author = {Taesup Moon and Georges Dupret and Shihao Ji and Ciya Liao and Zhaohui Zheng},
  title = {{User behavior driven ranking without editorial judgments}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2010},
  doi = {10.1145/1871437.1871650},
}

Loading related content...

Figure