Paving the Way to a Large-scale Pseudosense-annotated Dataset

Mohammad Taher Pilehvar, Roberto NavigliNAACL 2013

TL;DR

This paper introduces a novel method for generating "pseudosense-annotated datasets" by creating artificial words (pseudosense words) that mimic real ambiguous words. This approach effectively addresses the long-standing challenges of semantic awareness and broad coverage, enabling the creation of large-scale datasets crucial for advancing research in lexical semantics.

What Problem Does This Paper Address?

A significant bottleneck in computational linguistics is the scarcity of manually annotated data, particularly for tasks like Word Sense Disambiguation (WSD). While artificial "pseudosense-annotated datasets" offer a cost-effective solution, existing methods struggle with two key issues:

  1. Semantic Awareness: Pseudosenses need to reflect actual semantic relationships between senses of a real word, not just random word pairings (homonymy).
  2. Coverage: The ability to generate a vast number of diverse pseudowords that model a wide range of ambiguous words is limited.

Previous approaches, like the "vicinity-based" method, could achieve good semantic awareness but suffered from poor coverage, especially when requiring a minimum number of occurrences for each pseudosense. This paper aims to overcome these limitations to facilitate large-scale WSD research.

Methods

The paper proposes a similarity-based pseudoword generation approach that leverages the entire WordNet semantic network. Unlike the "vicinity-based" method, which only considers local subgraphs, this new approach uses Personalized PageRank (PPR).

Here's how it works:

  • Graph Representation: WordNet is treated as a graph where synsets are nodes and semantic relationships are edges.
  • Personalized PageRank (PPR): For each sense of a target ambiguous word, PPR is run. This algorithm computes a probability distribution, essentially measuring the "importance" or semantic similarity of all other synsets in WordNet to the starting sense.
  • Pseudosense Selection: The algorithm sorts the synsets based on their PPR scores. It then searches for monosemous (single-sense) words within these ranked synsets. A candidate word is selected as a pseudosense if it meets a minimum occurrence frequency threshold (minFreq) in a given corpus.
  • Coverage and Awareness: By exploring the entire WordNet graph, this method significantly expands the search space, addressing the coverage issue. The use of PPR inherently captures semantic similarity, thus promoting semantic awareness.

Datasets

  • WordNet 3.0: Used as the primary lexical resource for identifying ambiguous nouns and their senses.
  • English Gigaword corpus: Utilized to determine the occurrence frequency of words and to generate the actual pseudosense-annotated sentences.
  • Senseval-3 English lexical sample dataset: Used for evaluating the disambiguation difficulty of pseudowords. This dataset contains manually sense-tagged sentences for 20 ambiguous nouns.

The paper also introduces a large dataset of 15,935 pseudowords modeled after ambiguous nouns in WordNet, with the potential to generate at least 1000 annotated sentences per pseudosense.

Key Insights and Ideas

  • PPR as a Semantic Bridge: Personalized PageRank proves to be a powerful tool for capturing nuanced semantic similarity across the entire WordNet graph, going beyond local neighborhood analysis.
  • Balancing Coverage and Awareness: The similarity-based approach effectively balances the need for broad coverage (modeling many words) with semantic awareness (modeling real semantic relationships).
  • MinFreq's Impact: Increasing the minimum frequency requirement (minFreq) for pseudosenses leads to a deeper search within the PPR rankings, potentially relying on more indirect semantic relations, but still yielding good representative pseudosenses on average.
  • Pseudowords as Realistic Proxies: The evaluations suggest that these generated pseudowords are strong proxies for real ambiguous words, performing comparably in WSD tasks and preserving sense distinguishability.

Key Results

  • Coverage: The similarity-based approach achieves near-complete coverage (over 80% for minFreq=0, and 25% even for minFreq=1000) of ambiguous nouns in WordNet, a significant improvement over the vicinity-based method (which covered only 5 out of 20 nouns in a specific test).
  • Disambiguation Difficulty: WSD systems showed significantly better performance (75.14% recall) on similarity-based pseudowords compared to random pseudowords (78.80% recall), and this performance was much closer to the real word performance (73.26% recall). The similarity-based approach also demonstrated a closer recall to real data for 15 out of 20 tested nouns.
  • Representativeness: An average representativeness score of 3.12 (on a scale of 1-4) indicates that most pseudosenses are considered good substitutes for their real senses, even when derived from more distant semantic relations.
  • Distinguishability: The generated pseudowords effectively preserve the distinguishability of senses, with an overall score of 0.79 (on a scale of 0-1), meaning they don't tend to over-generalize and cover multiple senses.

Limitations and Future Work

  • Limitations:
    • The paper focuses primarily on nominal pseudowords, leaving extension to other parts of speech for future work.
  • While the similarity-based approach significantly improves coverage, the representativeness score can decrease slightly for higher polysemy degrees or when relying on more indirect semantic relations (as indicated by Table 5 and 7).
  • Future Work:
    • Extend the pseudoword generation technique to other parts of speech (verbs, adjectives, adverbs).
  • Utilize the generated pseudosense-tagged dataset for in-depth studies of various WSD paradigms.

Summary updated

BibTeX
@inproceedings{pilehvar2013paving,
  author = {Mohammad Taher Pilehvar and Roberto Navigli},
  title = {{Paving the Way to a Large-scale Pseudosense-annotated Dataset}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure