Automatic Identification of Infrequent Word Senses

Diana McCarthy, Rob Koeling, Julie Weeds, John CarrollCOLING 2004

TL;DR

This paper introduces an unsupervised method to automatically identify and filter out rare word senses from a generic lexicon like WordNet. By ranking senses based on their distributional and semantic similarity to neighboring words, the approach effectively prunes less frequent senses, making resources more domain-specific and efficient for NLP tasks like Word Sense Disambiguation (WSD).

What Problem Does This Paper Address?

Many NLP applications, particularly those dealing with word senses, face a significant challenge due to the inherent ambiguity of language. Words can have multiple meanings (senses), and the distribution of these senses is often highly skewed. This means that for many words, a few senses are very common, while a large number of senses are extremely rare. This skewness poses a problem for WSD systems, as rare senses are often underrepresented in training data and can lead to low recall or precision. Furthermore, when applying a generic lexicon like WordNet to a specific domain (e.g., medical texts, legal documents), many of its senses might be rare or even non-existent in that domain, leading to inefficiency and potential errors. The paper aims to address this by providing a method to automatically identify and filter out these infrequent or domain-irrelevant senses.

Methods

The core of the method involves two main components:

  1. Automatic Thesaurus Acquisition: The authors use Lin's (1998) distributional similarity measure to build a thesaurus. This thesaurus captures co-occurrence patterns of words based on grammatical relations (subject, direct object, noun-noun modification, adjective-noun modification) extracted from parsed text. For each word, the thesaurus lists its "nearest neighbors" ranked by their distributional similarity.
  2. Sense Ranking and Filtering: For a given target word, the method uses the automatically acquired thesaurus to rank its senses. The ranking score for each sense is calculated by summing the distributional similarity scores of its neighbors, weighted by the semantic similarity (using the WordNet Similarity Package, specifically the jcn measure) between the target sense and the sense of each neighbor. The intuition is that more prevalent senses will have more and more similar neighbors, leading to higher ranking scores. A threshold (Tw) is then applied to these scores, filtering out senses that fall below it. This threshold is set as a constant percentage of the score of the top-ranked sense, allowing for sensitivity to the specific word's ranking.

Datasets

  • BNC (British National Corpus): Used to acquire the thesaurus through automatic parsing and grammatical relation extraction. This is a large, balanced corpus of written English.
  • SemCor: A hand-tagged corpus used for evaluating the filtering method. It's a gold-standard dataset for WSD.
  • SENSEVAL-2 English all-words task: Another gold-standard dataset used for evaluation, comprising a smaller sample of hand-tagged text.
  • Reuters Corpus (SPORTS and FINANCE domains): Used to evaluate the method's ability to tailor sense inventories to domain-specific texts.
  • Subject Field Codes (SFC): Used as a resource for evaluating the domain-specific filtering, as it provides domain labels for WordNet synsets.

Key Insights and Ideas

  • Unsupervised Sense Filtering is Feasible: The paper demonstrates that it's possible to automatically identify and filter rare word senses without relying on hand-labeled data for the filtering process itself.
  • Distributional and Semantic Similarity are Powerful Indicators: Combining distributional similarity (from thesaurus neighbors) with semantic similarity (within WordNet) provides a robust way to estimate the prevalence of word senses.
  • Domain-Specific Tailoring is Crucial: Applying a generic lexicon like WordNet to specific domains often results in many irrelevant senses. The method effectively prunes these, making the lexicon more efficient and accurate for domain-specific tasks.
  • Erroneously Filtered Senses are Typically Rare: Even when the method makes mistakes and filters a sense that does occur in the test data, those senses tend to be less frequent than the ones that are kept. This suggests the filtering is generally conservative and targets genuinely rare senses.

Key Results

  • The method successfully identifies senses that do not occur in gold-standard datasets (SemCor and SENSEVAL-2).
  • For senses that do occur but are filtered, they typically have lower frequencies than the non-filtered senses.
  • In domain-specific experiments (SPORTS and FINANCE), the filtered senses were predominantly from outside the target domain, demonstrating the method's ability to tailor WordNet. For example, when filtering for the SPORTS domain, fewer "sports" labels were found among the filtered senses, and more "economy" and "commerce" senses were removed.
  • The percentage of filtered sense types varied significantly with the threshold (T%), allowing users to balance the number of filtered types against the risk of filtering correct senses.

Limitations and Future Work

  • Dependence on Thesaurus Quality: The performance of the filtering method is dependent on the quality of the automatically acquired thesaurus.
  • Choice of Similarity Measures: While jcn was used for semantic similarity, the authors suggest exploring other measures. The lesk measure, though less efficient, showed promise for other parts-of-speech (PoS) and for lexical resources without hierarchical structures.
  • Other Parts-of-Speech: The current work focuses on nouns. The authors plan to extend the ranking method to other PoS (adjectives, adverbs, verbs).
  • Application to WSD and Lexical Acquisition: The authors envision this method being directly useful for WSD systems to focus on more frequent senses and for lexical acquisition tasks.
  • Threshold Selection: The optimal threshold (Tw) can depend on the corpus and the application's requirements, suggesting a need for further investigation into threshold selection strategies.

Summary updated

BibTeX
@inproceedings{mccarthy2004automatic,
  author = {Diana McCarthy and Rob Koeling and Julie Weeds and John Carroll},
  title = {{Automatic Identification of Infrequent Word Senses}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2004},
}
Figure