A Comparative Study of Word Co-occurrence for Term Clustering in Language Model-based Sentence Retrieval
Lists
Only you see this
Abstract
Sentence retrieval is a very important part of question answering systems. Term clustering, in turn, is an effective approach for improving sentence retrieval performance: the more similar the terms in each cluster, the better the performance of the retrieval system. A key step in obtaining appropriate word clusters is accurate estimation of pairwise word similarities, based on their tendency to co-occur in similar contexts. In this paper, we compare four different methods for estimating word co-occurrence frequencies from two different corpora. The results show that different, commonly-used contexts for defining word co-occurrence differ significantly in retrieval performance. Using an appropriate co-occurrence criterion and corpus is shown to improve the mean average precision of sentence retrieval form 36.8% to 42.1%.
BibTeX
@inproceedings{momtazi2010comparative,
author = {Saeedeh Momtazi and Sanjeev Khudanpur and Dietrich Klakow},
title = {{A Comparative Study of Word Co-occurrence for Term Clustering in Language Model-based Sentence Retrieval}},
booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
year = {2010},
}