Comparison of Two Interactive Search Refinement Techniques

Olga Vechtomova, Murat KaramuftuogluNAACL 2004

TL;DR

This paper explores two interactive search refinement techniques for improving information retrieval. The authors found that expanding queries using noun phrases extracted from relevant documents (Method 2) yielded significantly better results than expanding based on selected sentences (Method 1), suggesting that richer contextual clues from noun phrases are more effective for query refinement.

What Problem Does This Paper Address?

Traditional information retrieval systems often struggle to precisely match user queries to relevant documents. Query expansion, a technique that adds terms to an original query, is a common method to improve retrieval accuracy. However, interactive query expansion, where users provide feedback to guide the expansion, can be time-consuming and requires careful design to be effective. The authors address the challenge of designing efficient and effective interactive query expansion methods that minimize user effort while maximizing retrieval performance, particularly within the context of the TREC HARD track, which emphasizes high accuracy retrieval and utilizes contextual metadata.

Methods

The paper presents and compares two interactive query expansion techniques:

  • Method 1: Sentence Selection: Users are presented with sentences from top-ranked documents that contain query terms. They select sentences they believe represent relevant documents. The documents associated with these selected sentences are then used for query expansion. The sentence selection algorithm prioritizes sentences with more query terms and higher term weights, normalizing for sentence length.
  • Method 2: Noun Phrase Selection: Users are shown noun phrases extracted from top-ranked documents. They select phrases that might contain useful terms for query expansion. These selected phrases are then split into individual terms to expand the original query. This method leverages part-of-speech tagging and noun phrase chunking to identify these phrases.

Both methods were evaluated within the TREC HARD track framework, which provided additional metadata like genre, purpose, and user familiarity to aid retrieval.

Datasets

The evaluation was conducted using the TREC HARD track corpus, which consists of 372,219 documents from newswire (New York Times, Associated Press Worldstream, Xinghua English) and governmental (Congressional Record, Federal Register) sources, totaling 1.7 GB. The authors also mention using past TREC data (Financial Times and Los Angeles Times newswire corpora) for initial testing and parameter tuning.

Key Insights and Ideas

  • Noun Phrases Offer Richer Context: The core insight is that noun phrases, when presented to users for selection, provide more effective contextual clues for query expansion than individual sentences. This suggests that the immediate linguistic neighbors of terms within noun phrases are more indicative of their intended meaning and relevance.
  • Minimizing User Effort is Crucial: Both methods aim to reduce the amount of text users need to read by presenting focused snippets (sentences or noun phrases) rather than full documents. This is essential for making interactive feedback practical.
  • Metadata's Role: The TREC HARD track's contextual metadata (genre, purpose, familiarity) was utilized, though its direct impact on the comparison of the two methods is less emphasized than the methods themselves.
  • Sentence Selection Performance: Method 1 (sentence selection) did not perform as well as expected, even showing no improvement over the baseline in the official TREC run. This was surprising given positive results in prior training runs. The authors speculate that differences in corpus (governmental documents in HARD vs. newswire in training) might be a factor.
  • Noun Phrase Expansion's Promise: Method 2 (noun phrase selection) demonstrated statistically significant improvements over the baseline, outperforming Method 1. This suggests its potential for more robust query refinement.
  • Impact of Feedback Document Count: Experiments showed that increasing the number of relevant documents for feedback generally improves average precision, but the gains diminish after a certain point. Crucially, adding a few non-relevant documents can significantly degrade performance, but this degradation plateaus if there are enough relevant documents.

Key Results

  • Method 2 (Noun Phrase Expansion) Outperformed Method 1 (Sentence Selection):
    • UWAThard3 (Method 2) achieved an 18% increase in average precision (SOFT-DOC evaluation) and a 26.4% increase (HARD-DOC evaluation) over the baseline (UWAThard1).
  • UWAThard2 (Method 1) showed no improvement over the baseline in the official TREC run, a surprising result.
  • All Submitted Runs Performed Well: The paper states that all their submitted runs were above the median in all evaluation measures.
  • Effect of Relevant Documents: Average precision increased with more relevant documents, up to 4, after which gains slowed.
  • Effect of Non-Relevant Documents: A few non-relevant documents caused a drop in AveP, but this effect stabilized if sufficient relevant documents were present.

Limitations and Future Work

  • Limitations:
    • The performance of Method 1 (sentence selection) was unexpectedly poor in the official TREC run, and the reasons are not fully understood, with corpus differences being a potential factor.
  • The authors acknowledge that determining the relevance of a document based on a single sentence can be difficult, especially for subtle aspects of a topic.
  • The study did not deeply explore the impact of extra-linguistic information (user familiarity, genre preferences) beyond its use in the HARD track framework.
  • Future Work:
    • Further investigation into the factors influencing the performance of both methods is needed.
  • The authors plan to address the challenge of incorporating contextual and extra-linguistic information more deeply, requiring more complex linguistic and stylistic analysis techniques.
  • Exploring more sophisticated methods for harnessing user familiarity, search purpose, and genre preferences is a key future direction.

Summary updated

BibTeX
@inproceedings{vechtomova2004comparison,
  author = {Olga Vechtomova and Murat Karamuftuoglu},
  title = {{Comparison of Two Interactive Search Refinement Techniques}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure