A Comparative Study of Word Co-occurrence for Term Clustering in Language Model-based Sentence Retrieval

Saeedeh Momtazi, Sanjeev Khudanpur, Dietrich Klakow

Abstract

In Question Answering (QA) systems, sentence retrieval often suffers from data sparsity due to the short length of text segments. Class-based language models address this by clustering semantically similar words. This paper presents a systematic evaluation of how different definitions of word co-occurrence—ranging from document-wise to syntactic relationships—affect the quality of clusters for sentence retrieval. Our findings reveal that tight local contexts, specifically 2-word windows, provide the most effective clusters. Furthermore, we demonstrate that the scale of training data is a critical factor: clusters derived from the massive Google n-grams corpus significantly outperform those from smaller, domain-specific datasets.

Introduction

Sentence retrieval is a core component of modern Question Answering (QA) systems. Unlike traditional document retrieval, searching over sentences introduces significant challenges: a single sentence rarely contains enough context to bridge the vocabulary gap between a query and a relevant result (e.g., matching "beverage" to "drink"). To mitigate this, researchers employ term clustering to group words with similar distributions. However, the definition of "context" used to compute these distributions varies widely. This study aims to determine which co-occurrence definition is most effective for the specific task of sentence-level retrieval.

Methodology

We utilize a class-based unigram language model for retrieval and the standard Brown clustering algorithm to generate word groups. The Brown algorithm is a hierarchical clustering technique that merges clusters to maximize the average mutual information between adjacent clusters. We compare four primary co-occurrence definitions:

  1. Document-wise: Words appearing anywhere in the same document.
  2. Sentence-wise: Words appearing within the same sentence.
  3. Window-wise: Words appearing within a fixed sliding window (sizes 2 and 5 were tested).
  4. Syntax-wise: Words sharing a direct parent-child dependency relationship in a parse tree (extracted via the MINIPAR parser).

Data and Experimental Setup

Clusters were trained on two distinct corpora: AQUAINT1 (English newswire) and Google n-grams (web-scale data). Evaluation was performed using the TREC 2006 QA track data, with sentence-level relevance judgments. The scale of the training data for each method is summarized in the table below.

Table 1: Statistics of word pairs extracted from different corpora and co-occurrence methods.

Results and Discussion

Our experiments show that local context is paramount for sentence retrieval. As seen in Figure 2(a), broader contexts like document-wise co-occurrence or a 5-word window performed poorly, yielding results close to the baseline. The tightest context—a 2-word window (window-2)—captured the most relevant similarities for this task. Additionally, syntax-wise co-occurrence significantly outperformed sentence-wise methods, confirming that syntactic relationships are stronger indicators of semantic similarity than mere proximity within a sentence.

Figure 2: (a) MAP vs. number of clusters for various co-occurrence methods on AQUAINT1. (b) Comparison of performance between AQUAINT1 and Google n-gram clusters.

A key finding is the impact of data volume. Figure 2(b) illustrates that clusters trained on the massive Google n-grams corpus consistently outperformed those from the domain-matched AQUAINT1 corpus, despite the web data being out-of-domain. The sheer volume of data in the Google corpus (over 328 million word pairs for window-2) allows for more robust cluster formation that is less sensitive to the specific number of clusters chosen. The best overall result was achieved using Google n-gram window-2 clusters with 100 clusters, reaching a Mean Average Precision (MAP) of 42.1%, compared to the word-based baseline of 36.8%.

Conclusion

This study demonstrates that for sentence retrieval, "narrower" context definitions like bigram windows or syntactic dependencies provide more useful semantic clusters than broad document-level co-occurrence. Furthermore, the benefits of massive data scale outweigh the advantages of domain specificity. These insights suggest that future improvements could be found by applying syntactic clustering to web-scale corpora.

Summary updated

Abstract

Sentence retrieval is a very important part of question answering systems. Term clustering, in turn, is an effective approach for improving sentence retrieval performance: the more similar the terms in each cluster, the better the performance of the retrieval system. A key step in obtaining appropriate word clusters is accurate estimation of pairwise word similarities, based on their tendency to co-occur in similar contexts. In this paper, we compare four different methods for estimating word co-occurrence frequencies from two different corpora. The results show that different, commonly-used contexts for defining word co-occurrence differ significantly in retrieval performance. Using an appropriate co-occurrence criterion and corpus is shown to improve the mean average precision of sentence retrieval form 36.8% to 42.1%.
BibTeX
@inproceedings{momtazi2010comparative,
  author = {Saeedeh Momtazi and Sanjeev Khudanpur and Dietrich Klakow},
  title = {{A Comparative Study of Word Co-occurrence for Term Clustering in Language Model-based Sentence Retrieval}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure