Representative & Informative Query Selection for Learning to Rank using Submodular Functions
Lists
Only you see this
Abstract
The performance of Learning to Rank algorithms strongly depend on the number of labelled queries in the training set, while the cost incurred in annotating a large number of queries with relevance judgements is prohibitively high. As a result, constructing such a training dataset involves selecting a set of candidate queries for labelling. In this work, we investigate query selection strategies for learning to rank aimed at actively selecting unlabelled queries to be labelled so as to minimize the data annotation cost. %total number of labelled queries -- without degrading the ranking performance. In particular, we characterize query selection based on two aspects of \emph{informativeness} and \emph{representativeness} and propose two novel query selection strategies (i) Permutation Probability based query selection and (ii) Topic Model based query selection which capture the two aspects, respectively. We further argue that an ideal query selection strategy should take into account both these aspects and as our final contribution, we present a submodular objective that couples both these aspects while selecting query subsets. We evaluate the quality of the proposed strategies on three real world learning to rank datasets and show that the proposed query selection methods results in significant performance gains compared to the existing state-of-the-art approaches.
BibTeX
@inproceedings{mehrotra2015representative,
author = {Rishabh Mehrotra and Emine Yilmaz},
title = {{Representative \& Informative Query Selection for Learning to Rank using Submodular Functions}},
booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {2015},
url = {https://openreview.net/forum?id=ryW7xUZO-r},
}Loading related content...