Evaluation of Automatically Identified Index Terms for Browsing Electronic Documents

Nina Wacholder, Judith L. Klavans, David K. EvansNAACL 2000

TL;DR

This paper evaluates three methods for automatically identifying index terms in documents to improve browsing. The authors found that a "Head Sorting" method, which focuses on the head of noun phrases, offers the best balance between the quality of terms and their coverage of a document's content, outperforming traditional keyword counting and technical term extraction. This work is crucial for building more effective document retrieval and summarization systems.

What Problem Does This Paper Address?

In the age of vast digital libraries, efficiently finding relevant information within documents is a major challenge. Traditional keyword indexing often relies on simple word frequency counts, which can be noisy and miss nuanced meanings. More sophisticated methods exist, but they often require domain-specific knowledge or complex lexicons, limiting their general applicability. This paper tackles the problem of identifying "index terms" – natural language expressions that meaningfully represent a document's content – using domain-independent methods. The goal is to create a "profile" or "thumbnail" of a document that humans can easily understand for browsing and information retrieval.

Methods

The researchers compared three domain-independent methods for extracting index terms:

  • Keywords (KW): This is a standard approach that identifies terms by counting the frequency of stemmed words within a document.
  • Technical Terms (TT): This method, based on Justeson and Katz (1995), identifies noun phrases (NPs) or sub-parts of NPs that appear more than twice in a document.
  • Head Sorted Terms (HS): This novel method, proposed by Wacholder (1998), focuses on "simplex noun phrases" (SNPs) – essentially the core noun phrase without complex post-modifiers. These SNPs are then sorted by their "head" word (typically the last word in the phrase), based on the linguistic principle that the head carries more semantic weight.

The evaluation was qualitative, involving human judges (professionals and students) who rated the usefulness of identified terms for indexing an article on a scale of 1 to 5. The researchers also considered two key criteria: quality (how good the term is as an index term) and coverage (how well the terms represent the document's significant topics).

Datasets

The study used three articles from the 1990 Wall Street Journal, each approximately 500 words long, from the Tipster collection of documents. These articles were processed using the three methods described above.

Key Insights and Ideas

  • The "Head" Matters: The core idea behind the Head Sorting (HS) method is that the head of a noun phrase is linguistically more significant than its modifiers. This principle, when applied to document indexing, seems to capture important concepts effectively.
  • Quality vs. Coverage Trade-off: Technical Terms (TTs) were found to be of very high quality but offered poor coverage due to their low numbers. Conversely, simple Keywords (KWs) provided good coverage but were often vague and of lower quality.
  • HS Strikes a Balance: The Head Sorting (HS) method emerged as a strong contender because it managed to balance the quality of index terms with their coverage of the document's content.
  • Phrases vs. Single Words: The study found that phrases were often rated more highly than individual words, especially when those words appeared in isolation. This suggests that context provided by phrases is valuable for indexing.
  • Professional vs. Student Ratings: While both groups participated, professionals tended to provide more consistent and reliable ratings, suggesting that domain expertise can influence the perception of index term usefulness.

Key Results

  • Quality: Technical Terms (TTs) received the highest average quality ratings (1.79 on a 1-5 scale), followed by Head Sorted terms (HS) (2.89), and then Keywords (KW) (3.27).
  • Coverage: While not explicitly quantified in a single number, the paper implies that HS provided better coverage than TTs, and KWs provided the broadest coverage but with lower quality.
  • Overall Performance: When considering both quality and coverage, the Head Sorting (HS) method outperformed both KW and TT methods. For instance, 100% of TTs received a rating of 2 or better, but only 8 TTs did so. In contrast, 41 HS terms received a rating of 2 or better.
  • Number of Terms: TTs were significantly fewer (average of 7 per 500 words) compared to HS and KW (over 50).

Limitations and Future Work

  • Subjectivity of Ratings: The study relied on human judges, which inherently introduces subjectivity. While professionals showed more consistency, the small number of professionals limited the statistical power of their ratings.
  • Presentation Bias: The authors acknowledge the challenge of presenting lists of varying lengths and made specific decisions to balance them, which might have introduced some bias.
  • Future Work: The paper suggests that the HS approach offers a promising direction for creating document profiles. Further research could explore combining HS with other methods or refining the HS algorithm to further improve its performance. The authors also highlight the importance of understanding how single words are perceived differently in isolation versus within phrases, which has implications for designing more effective indexing tools.

Summary updated

BibTeX
@inproceedings{wacholder2000evaluation,
  author = {Nina Wacholder and Judith L. Klavans and David K. Evans},
  title = {{Evaluation of Automatically Identified Index Terms for Browsing Electronic Documents}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2000},
}
Figure