Abstract
In Question Answering (QA) systems, sentence retrieval often suffers from data sparsity due to the short length of text segments. Class-based language models address this by clustering semantically similar words. This paper presents a systematic evaluation of how different definitions of word co-occurrence—ranging from document-wise to syntactic relationships—affect the quality of clusters for sentence retrieval. Our findings reveal that tight local contexts, specifically 2-word windows, provide the most effective clusters. Furthermore, we demonstrate that the scale of training data is a critical factor: clusters derived from the massive Google n-grams corpus significantly outperform those from smaller, domain-specific datasets.
Introduction
Sentence retrieval is a core component of modern Question Answering (QA) systems. Unlike traditional document retrieval, searching over sentences introduces significant challenges: a single sentence rarely contains enough context to bridge the vocabulary gap between a query and a relevant result (e.g., matching "beverage" to "drink"). To mitigate this, researchers employ term clustering to group words with similar distributions. However, the definition of "context" used to compute these distributions varies widely. This study aims to determine which co-occurrence definition is most effective for the specific task of sentence-level retrieval.
Methodology
We utilize a class-based unigram language model for retrieval and the standard Brown clustering algorithm to generate word groups. The Brown algorithm is a hierarchical clustering technique that merges clusters to maximize the average mutual information between adjacent clusters. We compare four primary co-occurrence definitions:
- Document-wise: Words appearing anywhere in the same document.
- Sentence-wise: Words appearing within the same sentence.
- Window-wise: Words appearing within a fixed sliding window (sizes 2 and 5 were tested).
- Syntax-wise: Words sharing a direct parent-child dependency relationship in a parse tree (extracted via the MINIPAR parser).
Data and Experimental Setup
Clusters were trained on two distinct corpora: AQUAINT1 (English newswire) and Google n-grams (web-scale data). Evaluation was performed using the TREC 2006 QA track data, with sentence-level relevance judgments. The scale of the training data for each method is summarized in the table below.
Results and Discussion
Our experiments show that local context is paramount for sentence retrieval. As seen in Figure 2(a), broader contexts like document-wise co-occurrence or a 5-word window performed poorly, yielding results close to the baseline. The tightest context—a 2-word window (window-2)—captured the most relevant similarities for this task. Additionally, syntax-wise co-occurrence significantly outperformed sentence-wise methods, confirming that syntactic relationships are stronger indicators of semantic similarity than mere proximity within a sentence.
A key finding is the impact of data volume. Figure 2(b) illustrates that clusters trained on the massive Google n-grams corpus consistently outperformed those from the domain-matched AQUAINT1 corpus, despite the web data being out-of-domain. The sheer volume of data in the Google corpus (over 328 million word pairs for window-2) allows for more robust cluster formation that is less sensitive to the specific number of clusters chosen. The best overall result was achieved using Google n-gram window-2 clusters with 100 clusters, reaching a Mean Average Precision (MAP) of 42.1%, compared to the word-based baseline of 36.8%.
Conclusion
This study demonstrates that for sentence retrieval, "narrower" context definitions like bigram windows or syntactic dependencies provide more useful semantic clusters than broad document-level co-occurrence. Furthermore, the benefits of massive data scale outweigh the advantages of domain specificity. These insights suggest that future improvements could be found by applying syntactic clustering to web-scale corpora.
Summary updated