Discriminative Power and Retrieval Effectiveness of Phrasal Indexing Terms

Sumio FujitaACL 2000

TL;DR

This paper investigates the effectiveness of using phrasal terms (multi-word expressions) for information retrieval in Japanese text. It finds that while phrasal terms can be good discriminators of relevant documents, their effectiveness is inconsistent and depends heavily on query length and how they are weighted, suggesting a need for more nuanced approaches beyond simple phrase inclusion.

What Problem Does This Paper Address?

The effectiveness of using phrasal terms (like "artificial intelligence" instead of just "artificial" and "intelligence") in information retrieval systems has been a long-standing debate. While intuitively phrases might capture concepts better than individual words, their actual impact on retrieval performance is not always clear and can be inconsistent. Recent studies suggest a correlation between query length and the benefit of using phrases. This paper aims to clarify this by analyzing the discriminative power and retrieval effectiveness of phrasal terms, particularly in the context of Japanese text, where morphological analysis is complex. The core challenge is to understand when and how phrasal terms contribute positively to retrieval, and when they might even be detrimental.

Methods

The authors employ a statistical approach to analyze phrasal terms extracted from a large Japanese text collection. They utilize concepts from information theory, specifically Mutual Information (MI), to quantify the relationship between a term (single word or phrase) and its relevance to a query.

  • Term Extraction: Phrasal terms are identified through linguistic processing of Japanese text.
  • Discriminative Power Analysis: They use p(occ|rel) (probability of a term occurring in relevant documents) and p(occ) (probability of a term occurring in any document) to analyze term distribution. They also introduce log(p(occ|rel)/p(occ)), which is related to mutual information, as a measure of a term's discriminative power.
  • Categorization of Phrasal Terms: Phrasal terms are categorized into three types based on their MI values relative to their constituent single words:
  • Informative: The phrase provides more information about relevance than its constituent words combined.
  • Neutral: The phrase provides some information, but not significantly more than its constituent words.
  • Destructive: The phrase provides less information or even negative information about relevance compared to its constituent words.
  • Retrieval Effectiveness Evaluation: They compare retrieval performance (measured by non-interpolated average precision) using single words versus single words plus supplemental phrasal terms across various query lengths and combinations of query fields. They also examine the correlation between the weight ratio of informative/destructive phrasal terms and performance differences.

Datasets

The primary dataset used is the NACSIS Test Collection 1 (NTCIR, 1999). This collection includes:

  • A large corpus of scientific paper abstracts (330,000 records, 590MB).
  • Topic descriptions with four fields: <title>, <description>, <narrative>, and <concepts>.
  • Relevance judgments for the topics.

The authors create 15 different query versions for each topic by combining these fields, allowing for analysis across varying query lengths.

Key Insights and Ideas

  • Phrasal terms are generally good discriminators, but not always effective: While phrasal terms often show a strong correlation with relevant documents, this doesn't automatically translate to improved retrieval performance.
  • Query length matters significantly: The effectiveness of phrasal indexing is highly dependent on query length. Longer queries, especially those combining multiple fields like <title>, <description>, and <narrative>, tend to benefit more from supplemental phrasal terms.
  • "Informative" phrases are key, but rare: The authors propose a strict definition of "informative" phrasal terms (those providing more information than their constituent words). While these terms are rare, they are crucial for positive impact.
  • Over-weighting can be detrimental: Supplemental phrasal terms, even if informative, can cause "topic deviation" or "query drift" if over-weighted. This can lead to degradation in retrieval performance, especially in shorter queries.
  • Zipfian distribution similarity: Phrasal terms, like single words, follow a Zipfian distribution, but their document frequencies are much lower, which can limit their usefulness despite good discriminative power.

Key Results

  • Inconsistent performance: Out of 15 query versions, phrasal indexing performed better than single-word indexing in 10 cases and worse in 5, highlighting its inconsistent nature.
  • Longer queries benefit more: Combining <title>, <description>, and <narrative> fields for queries showed improved performance with supplemental phrasal terms compared to single words.
  • Informative phrases are rare: Only about 30% of phrasal terms in short queries and 21% in long queries were classified as "informative" based on the authors' strict criteria.
  • Weak correlation with performance: The correlation between the weight ratio of informative/destructive phrasal terms and performance difference was observed to be very weak.
  • Degradation examples: The paper provides specific examples of phrasal terms that caused significant degradation in retrieval performance, even when classified as informative or neutral.

Limitations and Future Work

  • Inconsistent effectiveness: The primary limitation is the inconsistent effectiveness of phrasal indexing, which the authors acknowledge as a complex problem.
  • Weighting schemes: The current weighting schemes might not be optimal for phrasal terms, and the paper suggests that a new weighting scheme to replace empirical down-weighting for supplemental phrasal terms is needed.
  • Topic deviation: The phenomenon of "topic deviation" caused by over-weighted phrasal terms is a significant issue that requires further investigation.
  • Future work: The authors suggest this research is a starting point for a larger project aiming to develop better weighting schemes and a deeper understanding of phrasal term effectiveness. They also note that a different visualization might be more insightful for analyzing term distribution characteristics.

Summary updated

BibTeX
@inproceedings{fujita2000discriminative,
  author = {Sumio Fujita},
  title = {{Discriminative Power and Retrieval Effectiveness of Phrasal Indexing Terms}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2000},
}
Figure