Meta-scoring: Automatically Evaluating Term Weighting Schemes in IR without Precision-Recall

Rong Jin, Christos Falusos, Alexander G. Hauptmann

Abstract

In this paper, we present a method that can automatically evaluate performance of different term weighting schemes in information retrieval without resorting to precision-recall based on human relevance judgments. Specifically, the problem is: given two document-term matrixes generated from two different term weighting schemes, can we tell which term weighting scheme will performance better than the other? We propose a meta-scoring function, which takes as input the document-term matrix generated by some term weighting scheme and computes a goodness score from the document-term matrix. In our experiments, we found out that this score is highly correlated with the precision-recall measurement for all the collections and term weighting schema we tried. Thus, we conclude that our meta-scoring function can be a substitute for the precision-recall measurement that needs relevance judgments of human subject. Furthermore, this meta-scoring function is not limited only to text information retrieval can be applied to fields such as image and DNA retrieval.
BibTeX
@inproceedings{jin2001meta,
  author = {Rong Jin and Christos Falusos and Alexander G. Hauptmann},
  title = {{Meta-scoring: Automatically Evaluating Term Weighting Schemes in IR without Precision-Recall}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2001},
  doi = {10.1145/383952.383964},
}

Loading related content...

Figure